[llvm] r371429 - [Remarks] Add parser for bitstream remarks
Francis Visoiu Mistrih via llvm-commits
llvm-commits at lists.llvm.org
Mon Sep 9 10:43:50 PDT 2019
Author: thegameg
Date: Mon Sep 9 10:43:50 2019
New Revision: 371429
URL: http://llvm.org/viewvc/llvm-project?rev=371429&view=rev
Log:
[Remarks] Add parser for bitstream remarks
The bitstream remark serializer landed in r367372.
This adds a bitstream remark parser that parser bitstream remark files
to llvm::remarks::Remark objects through the RemarkParser interface.
A few interesting things to point out:
* There are parsing helpers to parse the different types of blocks
* The main parsing helper allows us to parse remark metadata and open an
external file containing the encoded remarks
* This adds a dependency from the Remarks library to the BitstreamReader
library
* The testing strategy is to create a remark entry through YAML, parse
it, serialize it to bitstream, parse that back and compare the objects.
* There are close to no tests for malformed bitstream remarks, due to
the lack of textual format for the bitstream format.
* This adds a new C API for parsing bitstream remarks:
LLVMRemarkParserCreateBitstream.
* This bumps the REMARKS_API_VERSION to 1.
Differential Revision: https://reviews.llvm.org/D67134
Added:
llvm/trunk/include/llvm/Remarks/BitstreamRemarkParser.h
llvm/trunk/lib/Remarks/BitstreamRemarkParser.cpp
llvm/trunk/lib/Remarks/BitstreamRemarkParser.h
llvm/trunk/unittests/Remarks/BitstreamRemarksParsingTest.cpp
Modified:
llvm/trunk/include/llvm-c/Remarks.h
llvm/trunk/include/llvm/Bitstream/BitstreamReader.h
llvm/trunk/lib/Remarks/CMakeLists.txt
llvm/trunk/lib/Remarks/LLVMBuild.txt
llvm/trunk/lib/Remarks/RemarkParser.cpp
llvm/trunk/tools/remarks-shlib/Remarks.exports
llvm/trunk/unittests/Remarks/CMakeLists.txt
Modified: llvm/trunk/include/llvm-c/Remarks.h
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm-c/Remarks.h?rev=371429&r1=371428&r2=371429&view=diff
==============================================================================
--- llvm/trunk/include/llvm-c/Remarks.h (original)
+++ llvm/trunk/include/llvm-c/Remarks.h Mon Sep 9 10:43:50 2019
@@ -30,7 +30,8 @@ extern "C" {
* @{
*/
-#define REMARKS_API_VERSION 0
+// 0 -> 1: Bitstream remarks support.
+#define REMARKS_API_VERSION 1
/**
* The type of the emitted remark.
@@ -241,6 +242,20 @@ extern LLVMRemarkParserRef LLVMRemarkPar
uint64_t Size);
/**
+ * Creates a remark parser that can be used to parse the buffer located in \p
+ * Buf of size \p Size bytes.
+ *
+ * \p Buf cannot be `NULL`.
+ *
+ * This function should be paired with LLVMRemarkParserDispose() to avoid
+ * leaking resources.
+ *
+ * \since REMARKS_API_VERSION=1
+ */
+extern LLVMRemarkParserRef LLVMRemarkParserCreateBitstream(const void *Buf,
+ uint64_t Size);
+
+/**
* Returns the next remark in the file.
*
* The value pointed to by the return value needs to be disposed using a call to
Modified: llvm/trunk/include/llvm/Bitstream/BitstreamReader.h
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Bitstream/BitstreamReader.h?rev=371429&r1=371428&r2=371429&view=diff
==============================================================================
--- llvm/trunk/include/llvm/Bitstream/BitstreamReader.h (original)
+++ llvm/trunk/include/llvm/Bitstream/BitstreamReader.h Mon Sep 9 10:43:50 2019
@@ -379,6 +379,7 @@ public:
using SimpleBitstreamCursor::ReadVBR;
using SimpleBitstreamCursor::ReadVBR64;
using SimpleBitstreamCursor::SizeInBytes;
+ using SimpleBitstreamCursor::skipToEnd;
/// Return the number of bits used to encode an abbrev #.
unsigned getAbbrevIDWidth() const { return CurCodeSize; }
Added: llvm/trunk/include/llvm/Remarks/BitstreamRemarkParser.h
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Remarks/BitstreamRemarkParser.h?rev=371429&view=auto
==============================================================================
--- llvm/trunk/include/llvm/Remarks/BitstreamRemarkParser.h (added)
+++ llvm/trunk/include/llvm/Remarks/BitstreamRemarkParser.h Mon Sep 9 10:43:50 2019
@@ -0,0 +1,116 @@
+//===-- BitstreamRemarkParser.h - Bitstream parser --------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file provides an implementation of the remark parser using the LLVM
+// Bitstream format.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_REMARKS_BITSTREAM_REMARK_PARSER_H
+#define LLVM_REMARKS_BITSTREAM_REMARK_PARSER_H
+
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Bitstream/BitstreamReader.h"
+#include "llvm/Remarks/BitstreamRemarkContainer.h"
+#include "llvm/Remarks/Remark.h"
+#include "llvm/Remarks/RemarkParser.h"
+#include "llvm/Support/Error.h"
+#include <array>
+
+namespace llvm {
+namespace remarks {
+
+/// Helper to parse a META_BLOCK for a bitstream remark container.
+struct BitstreamMetaParserHelper {
+ /// The Bitstream reader.
+ BitstreamCursor &Stream;
+ /// Reference to the storage for the block info.
+ BitstreamBlockInfo &BlockInfo;
+ /// The parsed content: depending on the container type, some fields might be
+ /// empty.
+ Optional<uint64_t> ContainerVersion;
+ Optional<uint8_t> ContainerType;
+ Optional<StringRef> StrTabBuf;
+ Optional<StringRef> ExternalFilePath;
+ Optional<uint64_t> RemarkVersion;
+
+ /// Continue parsing with \p Stream. \p Stream is expected to contain a
+ /// ENTER_SUBBLOCK to the META_BLOCK at the current position.
+ /// \p Stream is expected to have a BLOCKINFO_BLOCK set.
+ BitstreamMetaParserHelper(BitstreamCursor &Stream,
+ BitstreamBlockInfo &BlockInfo);
+
+ /// Parse the META_BLOCK and fill the available entries.
+ /// This helper does not check for the validity of the fields.
+ Error parse();
+};
+
+/// Helper to parse a REMARK_BLOCK for a bitstream remark container.
+struct BitstreamRemarkParserHelper {
+ /// The Bitstream reader.
+ BitstreamCursor &Stream;
+ /// The parsed content: depending on the remark, some fields might be empty.
+ Optional<uint8_t> Type;
+ Optional<uint64_t> RemarkNameIdx;
+ Optional<uint64_t> PassNameIdx;
+ Optional<uint64_t> FunctionNameIdx;
+ Optional<uint64_t> SourceFileNameIdx;
+ Optional<uint32_t> SourceLine;
+ Optional<uint32_t> SourceColumn;
+ Optional<uint64_t> Hotness;
+ struct Argument {
+ Optional<uint64_t> KeyIdx;
+ Optional<uint64_t> ValueIdx;
+ Optional<uint64_t> SourceFileNameIdx;
+ Optional<uint32_t> SourceLine;
+ Optional<uint32_t> SourceColumn;
+ };
+ Optional<ArrayRef<Argument>> Args;
+ /// Avoid re-allocating a vector every time.
+ SmallVector<Argument, 8> TmpArgs;
+
+ /// Continue parsing with \p Stream. \p Stream is expected to contain a
+ /// ENTER_SUBBLOCK to the REMARK_BLOCK at the current position.
+ /// \p Stream is expected to have a BLOCKINFO_BLOCK set and to have already
+ /// parsed the META_BLOCK.
+ BitstreamRemarkParserHelper(BitstreamCursor &Stream);
+
+ /// Parse the REMARK_BLOCK and fill the available entries.
+ /// This helper does not check for the validity of the fields.
+ Error parse();
+};
+
+/// Helper to parse any bitstream remark container.
+struct BitstreamParserHelper {
+ /// The Bitstream reader.
+ BitstreamCursor Stream;
+ /// The block info block.
+ BitstreamBlockInfo BlockInfo;
+ /// Start parsing at \p Buffer.
+ BitstreamParserHelper(StringRef Buffer);
+ /// Parse the magic number.
+ Expected<std::array<char, 4>> parseMagic();
+ /// Parse the block info block containing all the abbrevs.
+ /// This needs to be called before calling any other parsing function.
+ Error parseBlockInfoBlock();
+ /// Return true if the next block is a META_BLOCK. This function does not move
+ /// the cursor.
+ Expected<bool> isMetaBlock();
+ /// Return true if the next block is a REMARK_BLOCK. This function does not
+ /// move the cursor.
+ Expected<bool> isRemarkBlock();
+ /// Return true if the parser reached the end of the stream.
+ bool atEndOfStream() { return Stream.AtEndOfStream(); }
+ /// Jump to the end of the stream, skipping everything.
+ void skipToEnd() { return Stream.skipToEnd(); }
+};
+
+} // end namespace remarks
+} // end namespace llvm
+
+#endif /* LLVM_REMARKS_BITSTREAM_REMARK_PARSER_H */
Added: llvm/trunk/lib/Remarks/BitstreamRemarkParser.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Remarks/BitstreamRemarkParser.cpp?rev=371429&view=auto
==============================================================================
--- llvm/trunk/lib/Remarks/BitstreamRemarkParser.cpp (added)
+++ llvm/trunk/lib/Remarks/BitstreamRemarkParser.cpp Mon Sep 9 10:43:50 2019
@@ -0,0 +1,586 @@
+//===- BitstreamRemarkParser.cpp ------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file provides utility methods used by clients that want to use the
+// parser for remark diagnostics in LLVM.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Remarks/BitstreamRemarkParser.h"
+#include "BitstreamRemarkParser.h"
+#include "llvm/Remarks/BitstreamRemarkContainer.h"
+#include "llvm/Support/MemoryBuffer.h"
+
+using namespace llvm;
+using namespace llvm::remarks;
+
+static Error unknownRecord(const char *BlockName, unsigned RecordID) {
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing %s: unknown record entry (%lu).", BlockName,
+ RecordID);
+}
+
+static Error malformedRecord(const char *BlockName, const char *RecordName) {
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing %s: malformed record entry (%s).", BlockName,
+ RecordName);
+}
+
+BitstreamMetaParserHelper::BitstreamMetaParserHelper(
+ BitstreamCursor &Stream, BitstreamBlockInfo &BlockInfo)
+ : Stream(Stream), BlockInfo(BlockInfo) {}
+
+/// Parse a record and fill in the fields in the parser.
+static Error parseRecord(BitstreamMetaParserHelper &Parser, unsigned Code) {
+ BitstreamCursor &Stream = Parser.Stream;
+ // Note: 2 is used here because it's the max number of fields we have per
+ // record.
+ SmallVector<uint64_t, 2> Record;
+ StringRef Blob;
+ Expected<unsigned> RecordID = Stream.readRecord(Code, Record, &Blob);
+ if (!RecordID)
+ return RecordID.takeError();
+
+ switch (*RecordID) {
+ case RECORD_META_CONTAINER_INFO: {
+ if (Record.size() != 2)
+ return malformedRecord("BLOCK_META", "RECORD_META_CONTAINER_INFO");
+ Parser.ContainerVersion = Record[0];
+ Parser.ContainerType = Record[1];
+ break;
+ }
+ case RECORD_META_REMARK_VERSION: {
+ if (Record.size() != 1)
+ return malformedRecord("BLOCK_META", "RECORD_META_REMARK_VERSION");
+ Parser.RemarkVersion = Record[0];
+ break;
+ }
+ case RECORD_META_STRTAB: {
+ if (Record.size() != 0)
+ return malformedRecord("BLOCK_META", "RECORD_META_STRTAB");
+ Parser.StrTabBuf = Blob;
+ break;
+ }
+ case RECORD_META_EXTERNAL_FILE: {
+ if (Record.size() != 0)
+ return malformedRecord("BLOCK_META", "RECORD_META_EXTERNAL_FILE");
+ Parser.ExternalFilePath = Blob;
+ break;
+ }
+ default:
+ return unknownRecord("BLOCK_META", *RecordID);
+ }
+ return Error::success();
+}
+
+BitstreamRemarkParserHelper::BitstreamRemarkParserHelper(
+ BitstreamCursor &Stream)
+ : Stream(Stream) {}
+
+/// Parse a record and fill in the fields in the parser.
+static Error parseRecord(BitstreamRemarkParserHelper &Parser, unsigned Code) {
+ BitstreamCursor &Stream = Parser.Stream;
+ // Note: 5 is used here because it's the max number of fields we have per
+ // record.
+ SmallVector<uint64_t, 5> Record;
+ StringRef Blob;
+ Expected<unsigned> RecordID = Stream.readRecord(Code, Record, &Blob);
+ if (!RecordID)
+ return RecordID.takeError();
+
+ switch (*RecordID) {
+ case RECORD_REMARK_HEADER: {
+ if (Record.size() != 4)
+ return malformedRecord("BLOCK_REMARK", "RECORD_REMARK_HEADER");
+ Parser.Type = Record[0];
+ Parser.RemarkNameIdx = Record[1];
+ Parser.PassNameIdx = Record[2];
+ Parser.FunctionNameIdx = Record[3];
+ break;
+ }
+ case RECORD_REMARK_DEBUG_LOC: {
+ if (Record.size() != 3)
+ return malformedRecord("BLOCK_REMARK", "RECORD_REMARK_DEBUG_LOC");
+ Parser.SourceFileNameIdx = Record[0];
+ Parser.SourceLine = Record[1];
+ Parser.SourceColumn = Record[2];
+ break;
+ }
+ case RECORD_REMARK_HOTNESS: {
+ if (Record.size() != 1)
+ return malformedRecord("BLOCK_REMARK", "RECORD_REMARK_HOTNESS");
+ Parser.Hotness = Record[0];
+ break;
+ }
+ case RECORD_REMARK_ARG_WITH_DEBUGLOC: {
+ if (Record.size() != 5)
+ return malformedRecord("BLOCK_REMARK", "RECORD_REMARK_ARG_WITH_DEBUGLOC");
+ // Create a temporary argument. Use that as a valid memory location for this
+ // argument entry.
+ Parser.TmpArgs.emplace_back();
+ Parser.TmpArgs.back().KeyIdx = Record[0];
+ Parser.TmpArgs.back().ValueIdx = Record[1];
+ Parser.TmpArgs.back().SourceFileNameIdx = Record[2];
+ Parser.TmpArgs.back().SourceLine = Record[3];
+ Parser.TmpArgs.back().SourceColumn = Record[4];
+ Parser.Args =
+ ArrayRef<BitstreamRemarkParserHelper::Argument>(Parser.TmpArgs);
+ break;
+ }
+ case RECORD_REMARK_ARG_WITHOUT_DEBUGLOC: {
+ if (Record.size() != 2)
+ return malformedRecord("BLOCK_REMARK",
+ "RECORD_REMARK_ARG_WITHOUT_DEBUGLOC");
+ // Create a temporary argument. Use that as a valid memory location for this
+ // argument entry.
+ Parser.TmpArgs.emplace_back();
+ Parser.TmpArgs.back().KeyIdx = Record[0];
+ Parser.TmpArgs.back().ValueIdx = Record[1];
+ Parser.Args =
+ ArrayRef<BitstreamRemarkParserHelper::Argument>(Parser.TmpArgs);
+ break;
+ }
+ default:
+ return unknownRecord("BLOCK_REMARK", *RecordID);
+ }
+ return Error::success();
+}
+
+template <typename T>
+static Error parseBlock(T &ParserHelper, unsigned BlockID,
+ const char *BlockName) {
+ BitstreamCursor &Stream = ParserHelper.Stream;
+ Expected<BitstreamEntry> Next = Stream.advance();
+ if (!Next)
+ return Next.takeError();
+ if (Next->Kind != BitstreamEntry::SubBlock || Next->ID != BlockID)
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing %s: expecting [ENTER_SUBBLOCK, %s, ...].",
+ BlockName, BlockName);
+ if (Stream.EnterSubBlock(BlockID))
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while entering %s.", BlockName);
+
+ // Stop when there is nothing to read anymore or when we encounter an
+ // END_BLOCK.
+ while (!Stream.AtEndOfStream()) {
+ Expected<BitstreamEntry> Next = Stream.advance();
+ if (!Next)
+ return Next.takeError();
+ switch (Next->Kind) {
+ case BitstreamEntry::EndBlock:
+ return Error::success();
+ case BitstreamEntry::Error:
+ case BitstreamEntry::SubBlock:
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing %s: expecting records.", BlockName);
+ case BitstreamEntry::Record:
+ if (Error E = parseRecord(ParserHelper, Next->ID))
+ return E;
+ continue;
+ }
+ }
+ // If we're here, it means we didn't get an END_BLOCK yet, but we're at the
+ // end of the stream. In this case, error.
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing %s: unterminated block.", BlockName);
+}
+
+Error BitstreamMetaParserHelper::parse() {
+ return parseBlock(*this, META_BLOCK_ID, "META_BLOCK");
+}
+
+Error BitstreamRemarkParserHelper::parse() {
+ return parseBlock(*this, REMARK_BLOCK_ID, "REMARK_BLOCK");
+}
+
+BitstreamParserHelper::BitstreamParserHelper(StringRef Buffer)
+ : Stream(Buffer) {}
+
+Expected<std::array<char, 4>> BitstreamParserHelper::parseMagic() {
+ std::array<char, 4> Result;
+ for (unsigned i = 0; i < 4; ++i)
+ if (Expected<unsigned> R = Stream.Read(8))
+ Result[i] = *R;
+ else
+ return R.takeError();
+ return Result;
+}
+
+Error BitstreamParserHelper::parseBlockInfoBlock() {
+ Expected<BitstreamEntry> Next = Stream.advance();
+ if (!Next)
+ return Next.takeError();
+ if (Next->Kind != BitstreamEntry::SubBlock ||
+ Next->ID != llvm::bitc::BLOCKINFO_BLOCK_ID)
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing BLOCKINFO_BLOCK: expecting [ENTER_SUBBLOCK, "
+ "BLOCKINFO_BLOCK, ...].");
+
+ Expected<Optional<BitstreamBlockInfo>> MaybeBlockInfo =
+ Stream.ReadBlockInfoBlock();
+ if (!MaybeBlockInfo)
+ return MaybeBlockInfo.takeError();
+
+ if (!*MaybeBlockInfo)
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing BLOCKINFO_BLOCK.");
+
+ BlockInfo = **MaybeBlockInfo;
+
+ Stream.setBlockInfo(&BlockInfo);
+ return Error::success();
+}
+
+static Expected<bool> isBlock(BitstreamCursor &Stream, unsigned BlockID) {
+ bool Result = false;
+ uint64_t PreviousBitNo = Stream.GetCurrentBitNo();
+ Expected<BitstreamEntry> Next = Stream.advance();
+ if (!Next)
+ return Next.takeError();
+ switch (Next->Kind) {
+ case BitstreamEntry::SubBlock:
+ // Check for the block id.
+ Result = Next->ID == BlockID;
+ break;
+ case BitstreamEntry::Error:
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Unexpected error while parsing bitstream.");
+ default:
+ Result = false;
+ break;
+ }
+ if (Error E = Stream.JumpToBit(PreviousBitNo))
+ return std::move(E);
+ return Result;
+}
+
+Expected<bool> BitstreamParserHelper::isMetaBlock() {
+ return isBlock(Stream, META_BLOCK_ID);
+}
+
+Expected<bool> BitstreamParserHelper::isRemarkBlock() {
+ return isBlock(Stream, META_BLOCK_ID);
+}
+
+static Error validateMagicNumber(StringRef Magic) {
+ if (Magic != remarks::ContainerMagic)
+ return createStringError(std::make_error_code(std::errc::invalid_argument),
+ "Unknown magic number: expecting %s, got %.4s.",
+ remarks::ContainerMagic.data(), Magic.data());
+ return Error::success();
+}
+
+static Error advanceToMetaBlock(BitstreamParserHelper &Helper) {
+ Expected<std::array<char, 4>> Magic = Helper.parseMagic();
+ if (!Magic)
+ return Magic.takeError();
+ if (Error E = validateMagicNumber(StringRef(Magic->data(), Magic->size())))
+ return E;
+ if (Error E = Helper.parseBlockInfoBlock())
+ return E;
+ Expected<bool> isMetaBlock = Helper.isMetaBlock();
+ if (!isMetaBlock)
+ return isMetaBlock.takeError();
+ if (!*isMetaBlock)
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Expecting META_BLOCK after the BLOCKINFO_BLOCK.");
+ return Error::success();
+}
+
+Expected<std::unique_ptr<BitstreamRemarkParser>>
+remarks::createBitstreamParserFromMeta(StringRef Buf,
+ Optional<ParsedStringTable> StrTab) {
+ BitstreamParserHelper Helper(Buf);
+ Expected<std::array<char, 4>> Magic = Helper.parseMagic();
+ if (!Magic)
+ return Magic.takeError();
+
+ if (Error E = validateMagicNumber(StringRef(Magic->data(), Magic->size())))
+ return std::move(E);
+
+ return StrTab
+ ? std::make_unique<BitstreamRemarkParser>(Buf, std::move(*StrTab))
+ : std::make_unique<BitstreamRemarkParser>(Buf);
+}
+
+Expected<std::unique_ptr<Remark>> BitstreamRemarkParser::next() {
+ if (ParserHelper.atEndOfStream())
+ return make_error<EndOfFileError>();
+
+ if (!ReadyToParseRemarks) {
+ if (Error E = parseMeta())
+ return std::move(E);
+ ReadyToParseRemarks = true;
+ }
+
+ return parseRemark();
+}
+
+Error BitstreamRemarkParser::parseMeta() {
+ // Advance and to the meta block.
+ if (Error E = advanceToMetaBlock(ParserHelper))
+ return E;
+
+ BitstreamMetaParserHelper MetaHelper(ParserHelper.Stream,
+ ParserHelper.BlockInfo);
+ if (Error E = MetaHelper.parse())
+ return E;
+
+ if (Error E = processCommonMeta(MetaHelper))
+ return E;
+
+ switch (ContainerType) {
+ case BitstreamRemarkContainerType::Standalone:
+ return processStandaloneMeta(MetaHelper);
+ case BitstreamRemarkContainerType::SeparateRemarksFile:
+ return processSeparateRemarksFileMeta(MetaHelper);
+ case BitstreamRemarkContainerType::SeparateRemarksMeta:
+ return processSeparateRemarksMetaMeta(MetaHelper);
+ }
+}
+
+Error BitstreamRemarkParser::processCommonMeta(
+ BitstreamMetaParserHelper &MetaHelper) {
+ if (Optional<uint64_t> Version = MetaHelper.ContainerVersion)
+ ContainerVersion = *Version;
+ else
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing BLOCK_META: missing container version.");
+
+ if (Optional<uint8_t> Type = MetaHelper.ContainerType) {
+ if (*Type < static_cast<uint8_t>(BitstreamRemarkContainerType::First) ||
+ *Type > static_cast<uint8_t>(BitstreamRemarkContainerType::Last))
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing BLOCK_META: invalid container type.");
+
+ ContainerType = static_cast<BitstreamRemarkContainerType>(*Type);
+ } else
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing BLOCK_META: missing container type.");
+
+ return Error::success();
+}
+
+static Error processStrTab(BitstreamRemarkParser &P,
+ Optional<StringRef> StrTabBuf) {
+ if (!StrTabBuf)
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing BLOCK_META: missing string table.");
+ // Parse and assign the string table.
+ P.StrTab.emplace(*StrTabBuf);
+ return Error::success();
+}
+
+static Error processRemarkVersion(BitstreamRemarkParser &P,
+ Optional<uint64_t> RemarkVersion) {
+ if (!RemarkVersion)
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing BLOCK_META: missing remark version.");
+ P.RemarkVersion = *RemarkVersion;
+ return Error::success();
+}
+
+Error BitstreamRemarkParser::processExternalFilePath(
+ Optional<StringRef> ExternalFilePath) {
+ if (!ExternalFilePath)
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing BLOCK_META: missing external file path.");
+
+ // External file: open the external file, parse it, check if its metadata
+ // matches the one from the separate metadata, then replace the current parser
+ // with the one parsing the remarks.
+ ErrorOr<std::unique_ptr<MemoryBuffer>> BufferOrErr =
+ MemoryBuffer::getFile(*ExternalFilePath);
+ if (std::error_code EC = BufferOrErr.getError())
+ return errorCodeToError(EC);
+ TmpRemarkBuffer = std::move(*BufferOrErr);
+
+ // Create a separate parser used for parsing the separate file.
+ ParserHelper = BitstreamParserHelper(TmpRemarkBuffer->getBuffer());
+ // Advance and check until we can parse the meta block.
+ if (Error E = advanceToMetaBlock(ParserHelper))
+ return E;
+ // Parse the meta from the separate file.
+ // Note: here we overwrite the BlockInfo with the one from the file. This will
+ // be used to parse the rest of the file.
+ BitstreamMetaParserHelper SeparateMetaHelper(ParserHelper.Stream,
+ ParserHelper.BlockInfo);
+ if (Error E = SeparateMetaHelper.parse())
+ return E;
+
+ uint64_t PreviousContainerVersion = ContainerVersion;
+ if (Error E = processCommonMeta(SeparateMetaHelper))
+ return E;
+
+ if (ContainerType != BitstreamRemarkContainerType::SeparateRemarksFile)
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing external file's BLOCK_META: wrong container "
+ "type.");
+
+ if (PreviousContainerVersion != ContainerVersion)
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing external file's BLOCK_META: mismatching versions: "
+ "original meta: %lu, external file meta: %lu.",
+ PreviousContainerVersion, ContainerVersion);
+
+ // Process the meta from the separate file.
+ return processSeparateRemarksFileMeta(SeparateMetaHelper);
+}
+
+Error BitstreamRemarkParser::processStandaloneMeta(
+ BitstreamMetaParserHelper &Helper) {
+ if (Error E = processStrTab(*this, Helper.StrTabBuf))
+ return E;
+ return processRemarkVersion(*this, Helper.RemarkVersion);
+}
+
+Error BitstreamRemarkParser::processSeparateRemarksFileMeta(
+ BitstreamMetaParserHelper &Helper) {
+ return processRemarkVersion(*this, Helper.RemarkVersion);
+}
+
+Error BitstreamRemarkParser::processSeparateRemarksMetaMeta(
+ BitstreamMetaParserHelper &Helper) {
+ if (Error E = processStrTab(*this, Helper.StrTabBuf))
+ return E;
+ return processExternalFilePath(Helper.ExternalFilePath);
+}
+
+Expected<std::unique_ptr<Remark>> BitstreamRemarkParser::parseRemark() {
+ BitstreamRemarkParserHelper RemarkHelper(ParserHelper.Stream);
+ if (Error E = RemarkHelper.parse())
+ return std::move(E);
+
+ return processRemark(RemarkHelper);
+}
+
+Expected<std::unique_ptr<Remark>>
+BitstreamRemarkParser::processRemark(BitstreamRemarkParserHelper &Helper) {
+ std::unique_ptr<Remark> Result = std::make_unique<Remark>();
+ Remark &R = *Result;
+
+ if (StrTab == None)
+ return createStringError(
+ std::make_error_code(std::errc::invalid_argument),
+ "Error while parsing BLOCK_REMARK: missing string table.");
+
+ if (!Helper.Type)
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing BLOCK_REMARK: missing remark type.");
+
+ if (*Helper.Type < static_cast<uint8_t>(Type::First) ||
+ *Helper.Type > static_cast<uint8_t>(Type::Last))
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing BLOCK_REMARK: unknown remark type.");
+
+ R.RemarkType = static_cast<Type>(*Helper.Type);
+
+ if (!Helper.RemarkNameIdx)
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing BLOCK_REMARK: missing remark name.");
+
+ if (Expected<StringRef> RemarkName = (*StrTab)[*Helper.RemarkNameIdx])
+ R.RemarkName = *RemarkName;
+ else
+ return RemarkName.takeError();
+
+ if (!Helper.PassNameIdx)
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing BLOCK_REMARK: missing remark pass.");
+
+ if (Expected<StringRef> PassName = (*StrTab)[*Helper.PassNameIdx])
+ R.PassName = *PassName;
+ else
+ return PassName.takeError();
+
+ if (!Helper.FunctionNameIdx)
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing BLOCK_REMARK: missing remark function name.");
+ if (Expected<StringRef> FunctionName = (*StrTab)[*Helper.FunctionNameIdx])
+ R.FunctionName = *FunctionName;
+ else
+ return FunctionName.takeError();
+
+ if (Helper.SourceFileNameIdx && Helper.SourceLine && Helper.SourceColumn) {
+ Expected<StringRef> SourceFileName = (*StrTab)[*Helper.SourceFileNameIdx];
+ if (!SourceFileName)
+ return SourceFileName.takeError();
+ R.Loc.emplace();
+ R.Loc->SourceFilePath = *SourceFileName;
+ R.Loc->SourceLine = *Helper.SourceLine;
+ R.Loc->SourceColumn = *Helper.SourceColumn;
+ }
+
+ if (Helper.Hotness)
+ R.Hotness = *Helper.Hotness;
+
+ if (!Helper.Args)
+ return std::move(Result);
+
+ for (const BitstreamRemarkParserHelper::Argument &Arg : *Helper.Args) {
+ if (!Arg.KeyIdx)
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing BLOCK_REMARK: missing key in remark argument.");
+ if (!Arg.ValueIdx)
+ return createStringError(
+ std::make_error_code(std::errc::illegal_byte_sequence),
+ "Error while parsing BLOCK_REMARK: missing value in remark "
+ "argument.");
+
+ // We have at least a key and a value, create an entry.
+ R.Args.emplace_back();
+
+ if (Expected<StringRef> Key = (*StrTab)[*Arg.KeyIdx])
+ R.Args.back().Key = *Key;
+ else
+ return Key.takeError();
+
+ if (Expected<StringRef> Value = (*StrTab)[*Arg.ValueIdx])
+ R.Args.back().Val = *Value;
+ else
+ return Value.takeError();
+
+ if (Arg.SourceFileNameIdx && Arg.SourceLine && Arg.SourceColumn) {
+ if (Expected<StringRef> SourceFileName =
+ (*StrTab)[*Arg.SourceFileNameIdx]) {
+ R.Args.back().Loc.emplace();
+ R.Args.back().Loc->SourceFilePath = *SourceFileName;
+ R.Args.back().Loc->SourceLine = *Arg.SourceLine;
+ R.Args.back().Loc->SourceColumn = *Arg.SourceColumn;
+ } else
+ return SourceFileName.takeError();
+ }
+ }
+
+ return std::move(Result);
+}
Added: llvm/trunk/lib/Remarks/BitstreamRemarkParser.h
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Remarks/BitstreamRemarkParser.h?rev=371429&view=auto
==============================================================================
--- llvm/trunk/lib/Remarks/BitstreamRemarkParser.h (added)
+++ llvm/trunk/lib/Remarks/BitstreamRemarkParser.h Mon Sep 9 10:43:50 2019
@@ -0,0 +1,83 @@
+//===-- BitstreamRemarkParser.h - Parser for Bitstream remarks --*- C++/-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file provides the impementation of the Bitstream remark parser.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIB_REMARKS_BITSTREAM_REMARK_PARSER_H
+#define LLVM_LIB_REMARKS_BITSTREAM_REMARK_PARSER_H
+
+#include "llvm/ADT/Optional.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/Remarks/BitstreamRemarkParser.h"
+#include "llvm/Remarks/RemarkFormat.h"
+#include "llvm/Remarks/RemarkParser.h"
+#include "llvm/Support/raw_ostream.h"
+#include <memory>
+#include <string>
+
+namespace llvm {
+namespace remarks {
+/// Parses and holds the state of the latest parsed remark.
+struct BitstreamRemarkParser : public RemarkParser {
+ /// The buffer to parse.
+ BitstreamParserHelper ParserHelper;
+ /// The string table used for parsing strings.
+ Optional<ParsedStringTable> StrTab;
+ /// Temporary remark buffer used when the remarks are stored separately.
+ std::unique_ptr<MemoryBuffer> TmpRemarkBuffer;
+ /// The common metadata used to decide how to parse the buffer.
+ /// This is filled when parsing the metadata block.
+ uint64_t ContainerVersion;
+ uint64_t RemarkVersion;
+ BitstreamRemarkContainerType ContainerType;
+ /// Wether the parser is ready to parse remarks.
+ bool ReadyToParseRemarks = false;
+
+ /// Create a parser that expects to find a string table embedded in the
+ /// stream.
+ BitstreamRemarkParser(StringRef Buf)
+ : RemarkParser(Format::Bitstream), ParserHelper(Buf) {}
+
+ /// Create a parser that uses a pre-parsed string table.
+ BitstreamRemarkParser(StringRef Buf, ParsedStringTable StrTab)
+ : RemarkParser(Format::Bitstream), ParserHelper(Buf),
+ StrTab(std::move(StrTab)) {}
+
+ Expected<std::unique_ptr<Remark>> next() override;
+
+ static bool classof(const RemarkParser *P) {
+ return P->ParserFormat == Format::Bitstream;
+ }
+
+ /// Parse and process the metadata of the buffer.
+ Error parseMeta();
+
+ /// Parse a Bitstream remark.
+ Expected<std::unique_ptr<Remark>> parseRemark();
+
+private:
+ /// Helper functions.
+ Error processCommonMeta(BitstreamMetaParserHelper &Helper);
+ Error processStandaloneMeta(BitstreamMetaParserHelper &Helper);
+ Error processSeparateRemarksFileMeta(BitstreamMetaParserHelper &Helper);
+ Error processSeparateRemarksMetaMeta(BitstreamMetaParserHelper &Helper);
+ Expected<std::unique_ptr<Remark>>
+ processRemark(BitstreamRemarkParserHelper &Helper);
+ Error processExternalFilePath(Optional<StringRef> ExternalFilePath);
+};
+
+Expected<std::unique_ptr<BitstreamRemarkParser>>
+createBitstreamParserFromMeta(StringRef Buf,
+ Optional<ParsedStringTable> StrTab = None);
+
+} // end namespace remarks
+} // end namespace llvm
+
+#endif /* LLVM_LIB_REMARKS_BITSTREAM_REMARK_PARSER_H */
Modified: llvm/trunk/lib/Remarks/CMakeLists.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Remarks/CMakeLists.txt?rev=371429&r1=371428&r2=371429&view=diff
==============================================================================
--- llvm/trunk/lib/Remarks/CMakeLists.txt (original)
+++ llvm/trunk/lib/Remarks/CMakeLists.txt Mon Sep 9 10:43:50 2019
@@ -1,4 +1,5 @@
add_llvm_library(LLVMRemarks
+ BitstreamRemarkParser.cpp
BitstreamRemarkSerializer.cpp
Remark.cpp
RemarkFormat.cpp
Modified: llvm/trunk/lib/Remarks/LLVMBuild.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Remarks/LLVMBuild.txt?rev=371429&r1=371428&r2=371429&view=diff
==============================================================================
--- llvm/trunk/lib/Remarks/LLVMBuild.txt (original)
+++ llvm/trunk/lib/Remarks/LLVMBuild.txt Mon Sep 9 10:43:50 2019
@@ -18,4 +18,4 @@
type = Library
name = Remarks
parent = Libraries
-required_libraries = Support
+required_libraries = BitstreamReader Support
Modified: llvm/trunk/lib/Remarks/RemarkParser.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Remarks/RemarkParser.cpp?rev=371429&r1=371428&r2=371429&view=diff
==============================================================================
--- llvm/trunk/lib/Remarks/RemarkParser.cpp (original)
+++ llvm/trunk/lib/Remarks/RemarkParser.cpp Mon Sep 9 10:43:50 2019
@@ -12,6 +12,7 @@
//===----------------------------------------------------------------------===//
#include "llvm/Remarks/RemarkParser.h"
+#include "BitstreamRemarkParser.h"
#include "YAMLRemarkParser.h"
#include "llvm-c/Remarks.h"
#include "llvm/ADT/STLExtras.h"
@@ -57,8 +58,7 @@ llvm::remarks::createRemarkParser(Format
std::make_error_code(std::errc::invalid_argument),
"The YAML with string table format requires a parsed string table.");
case Format::Bitstream:
- return createStringError(std::make_error_code(std::errc::invalid_argument),
- "Parsing bitstream remarks is not supported.");
+ return std::make_unique<BitstreamRemarkParser>(Buf);
case Format::Unknown:
return createStringError(std::make_error_code(std::errc::invalid_argument),
"Unknown remark parser format.");
@@ -77,8 +77,7 @@ llvm::remarks::createRemarkParser(Format
case Format::YAMLStrTab:
return std::make_unique<YAMLStrTabRemarkParser>(Buf, std::move(StrTab));
case Format::Bitstream:
- return createStringError(std::make_error_code(std::errc::invalid_argument),
- "Parsing bitstream remarks is not supported.");
+ return std::make_unique<BitstreamRemarkParser>(Buf, std::move(StrTab));
case Format::Unknown:
return createStringError(std::make_error_code(std::errc::invalid_argument),
"Unknown remark parser format.");
@@ -96,8 +95,7 @@ llvm::remarks::createRemarkParserFromMet
case Format::YAMLStrTab:
return createYAMLParserFromMeta(Buf, std::move(StrTab));
case Format::Bitstream:
- return createStringError(std::make_error_code(std::errc::invalid_argument),
- "Parsing bitstream remarks is not supported.");
+ return createBitstreamParserFromMeta(Buf, std::move(StrTab));
case Format::Unknown:
return createStringError(std::make_error_code(std::errc::invalid_argument),
"Unknown remark parser format.");
@@ -132,6 +130,12 @@ extern "C" LLVMRemarkParserRef LLVMRemar
StringRef(static_cast<const char *>(Buf), Size)));
}
+extern "C" LLVMRemarkParserRef LLVMRemarkParserCreateBitstream(const void *Buf,
+ uint64_t Size) {
+ return wrap(new CParser(Format::Bitstream,
+ StringRef(static_cast<const char *>(Buf), Size)));
+}
+
extern "C" LLVMRemarkEntryRef
LLVMRemarkParserGetNext(LLVMRemarkParserRef Parser) {
CParser &TheCParser = *unwrap(Parser);
Modified: llvm/trunk/tools/remarks-shlib/Remarks.exports
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/tools/remarks-shlib/Remarks.exports?rev=371429&r1=371428&r2=371429&view=diff
==============================================================================
--- llvm/trunk/tools/remarks-shlib/Remarks.exports (original)
+++ llvm/trunk/tools/remarks-shlib/Remarks.exports Mon Sep 9 10:43:50 2019
@@ -17,6 +17,7 @@ LLVMRemarkEntryGetNumArgs
LLVMRemarkEntryGetFirstArg
LLVMRemarkEntryGetNextArg
LLVMRemarkParserCreateYAML
+LLVMRemarkParserCreateBitstream
LLVMRemarkParserGetNext
LLVMRemarkParserHasError
LLVMRemarkParserGetErrorMessage
Added: llvm/trunk/unittests/Remarks/BitstreamRemarksParsingTest.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/unittests/Remarks/BitstreamRemarksParsingTest.cpp?rev=371429&view=auto
==============================================================================
--- llvm/trunk/unittests/Remarks/BitstreamRemarksParsingTest.cpp (added)
+++ llvm/trunk/unittests/Remarks/BitstreamRemarksParsingTest.cpp Mon Sep 9 10:43:50 2019
@@ -0,0 +1,401 @@
+//===- unittests/Support/BitstreamRemarksParsingTest.cpp - Parsing tests --===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm-c/Remarks.h"
+#include "llvm/Remarks/Remark.h"
+#include "llvm/Remarks/RemarkParser.h"
+#include "llvm/Remarks/RemarkSerializer.h"
+#include "gtest/gtest.h"
+
+using namespace llvm;
+
+template <size_t N> void parseGood(const char (&Buf)[N]) {
+ // 1. Parse the YAML remark -> FromYAMLRemark
+ // 2. Serialize it to bitstream -> BSStream
+ // 3. Parse it back -> FromBSRemark
+ // 4. Compare the remark objects
+ //
+ // This testing methodology has the drawback of relying on both the YAML
+ // remark parser and the bitstream remark serializer. It does simplify
+ // testing a lot, since working directly with bitstream is not that easy.
+
+ // 1.
+ Expected<std::unique_ptr<remarks::RemarkParser>> MaybeParser =
+ remarks::createRemarkParser(remarks::Format::YAML, {Buf, N - 1});
+ EXPECT_FALSE(errorToBool(MaybeParser.takeError()));
+ EXPECT_TRUE(*MaybeParser != nullptr);
+
+ std::unique_ptr<remarks::Remark> FromYAMLRemark = nullptr;
+ remarks::RemarkParser &Parser = **MaybeParser;
+ Expected<std::unique_ptr<remarks::Remark>> Remark = Parser.next();
+ EXPECT_FALSE(errorToBool(Remark.takeError())); // Check for parsing errors.
+ EXPECT_TRUE(*Remark != nullptr); // At least one remark.
+ // Keep the previous remark around.
+ FromYAMLRemark = std::move(*Remark);
+ Remark = Parser.next();
+ Error E = Remark.takeError();
+ EXPECT_TRUE(E.isA<remarks::EndOfFileError>());
+ EXPECT_TRUE(errorToBool(std::move(E))); // Check for parsing errors.
+
+ // 2.
+ remarks::StringTable BSStrTab;
+ BSStrTab.internalize(*FromYAMLRemark);
+ std::string BSBuf;
+ raw_string_ostream BSStream(BSBuf);
+ Expected<std::unique_ptr<remarks::RemarkSerializer>> BSSerializer =
+ remarks::createRemarkSerializer(remarks::Format::Bitstream,
+ remarks::SerializerMode::Standalone,
+ BSStream, std::move(BSStrTab));
+ EXPECT_FALSE(errorToBool(BSSerializer.takeError()));
+ (*BSSerializer)->emit(*FromYAMLRemark);
+
+ // 3.
+ Expected<std::unique_ptr<remarks::RemarkParser>> MaybeBSParser =
+ remarks::createRemarkParser(remarks::Format::Bitstream, BSStream.str());
+ EXPECT_FALSE(errorToBool(MaybeBSParser.takeError()));
+ EXPECT_TRUE(*MaybeBSParser != nullptr);
+
+ std::unique_ptr<remarks::Remark> FromBSRemark = nullptr;
+ remarks::RemarkParser &BSParser = **MaybeBSParser;
+ Expected<std::unique_ptr<remarks::Remark>> BSRemark = BSParser.next();
+ EXPECT_FALSE(errorToBool(BSRemark.takeError())); // Check for parsing errors.
+ EXPECT_TRUE(*BSRemark != nullptr); // At least one remark.
+ // Keep the previous remark around.
+ FromBSRemark = std::move(*BSRemark);
+ BSRemark = BSParser.next();
+ Error BSE = BSRemark.takeError();
+ EXPECT_TRUE(BSE.isA<remarks::EndOfFileError>());
+ EXPECT_TRUE(errorToBool(std::move(BSE))); // Check for parsing errors.
+
+ EXPECT_EQ(*FromYAMLRemark, *FromBSRemark);
+}
+
+TEST(BitstreamRemarks, ParsingGood) {
+ parseGood("\n"
+ "--- !Missed\n"
+ "Pass: inline\n"
+ "Name: NoDefinition\n"
+ "DebugLoc: { File: file.c, Line: 3, Column: 12 }\n"
+ "Function: foo\n"
+ "Args:\n"
+ " - Callee: bar\n"
+ " - String: ' will not be inlined into '\n"
+ " - Caller: foo\n"
+ " DebugLoc: { File: file.c, Line: 2, Column: 0 }\n"
+ " - String: ' because its definition is unavailable'\n"
+ "");
+
+ // No debug loc should also pass.
+ parseGood("\n"
+ "--- !Missed\n"
+ "Pass: inline\n"
+ "Name: NoDefinition\n"
+ "Function: foo\n"
+ "Args:\n"
+ " - Callee: bar\n"
+ " - String: ' will not be inlined into '\n"
+ " - Caller: foo\n"
+ " DebugLoc: { File: file.c, Line: 2, Column: 0 }\n"
+ " - String: ' because its definition is unavailable'\n"
+ "");
+
+ // No args is also ok.
+ parseGood("\n"
+ "--- !Missed\n"
+ "Pass: inline\n"
+ "Name: NoDefinition\n"
+ "DebugLoc: { File: file.c, Line: 3, Column: 12 }\n"
+ "Function: foo\n"
+ "");
+}
+
+// Mandatory common part of a remark.
+#define COMMON_REMARK "\nPass: inline\nName: NoDefinition\nFunction: foo\n\n"
+// Test all the types.
+TEST(BitstreamRemarks, ParsingTypes) {
+ // Type: Passed
+ parseGood("--- !Passed" COMMON_REMARK);
+ // Type: Missed
+ parseGood("--- !Missed" COMMON_REMARK);
+ // Type: Analysis
+ parseGood("--- !Analysis" COMMON_REMARK);
+ // Type: AnalysisFPCommute
+ parseGood("--- !AnalysisFPCommute" COMMON_REMARK);
+ // Type: AnalysisAliasing
+ parseGood("--- !AnalysisAliasing" COMMON_REMARK);
+ // Type: Failure
+ parseGood("--- !Failure" COMMON_REMARK);
+}
+#undef COMMON_REMARK
+
+static inline StringRef checkStr(StringRef Str, unsigned ExpectedLen) {
+ const char *StrData = Str.data();
+ unsigned StrLen = Str.size();
+ EXPECT_EQ(StrLen, ExpectedLen);
+ return StringRef(StrData, StrLen);
+}
+
+TEST(BitstreamRemarks, Contents) {
+ StringRef Buf = "\n"
+ "--- !Missed\n"
+ "Pass: inline\n"
+ "Name: NoDefinition\n"
+ "DebugLoc: { File: file.c, Line: 3, Column: 12 }\n"
+ "Function: foo\n"
+ "Hotness: 4\n"
+ "Args:\n"
+ " - Callee: bar\n"
+ " - String: ' will not be inlined into '\n"
+ " - Caller: foo\n"
+ " DebugLoc: { File: file.c, Line: 2, Column: 0 }\n"
+ " - String: ' because its definition is unavailable'\n"
+ "\n";
+
+ Expected<std::unique_ptr<remarks::RemarkParser>> MaybeParser =
+ remarks::createRemarkParser(remarks::Format::YAML, Buf);
+ EXPECT_FALSE(errorToBool(MaybeParser.takeError()));
+ EXPECT_TRUE(*MaybeParser != nullptr);
+
+ remarks::RemarkParser &Parser = **MaybeParser;
+ Expected<std::unique_ptr<remarks::Remark>> MaybeRemark = Parser.next();
+ EXPECT_FALSE(
+ errorToBool(MaybeRemark.takeError())); // Check for parsing errors.
+ EXPECT_TRUE(*MaybeRemark != nullptr); // At least one remark.
+
+ const remarks::Remark &Remark = **MaybeRemark;
+ EXPECT_EQ(Remark.RemarkType, remarks::Type::Missed);
+ EXPECT_EQ(checkStr(Remark.PassName, 6), "inline");
+ EXPECT_EQ(checkStr(Remark.RemarkName, 12), "NoDefinition");
+ EXPECT_EQ(checkStr(Remark.FunctionName, 3), "foo");
+ EXPECT_TRUE(Remark.Loc);
+ const remarks::RemarkLocation &RL = *Remark.Loc;
+ EXPECT_EQ(checkStr(RL.SourceFilePath, 6), "file.c");
+ EXPECT_EQ(RL.SourceLine, 3U);
+ EXPECT_EQ(RL.SourceColumn, 12U);
+ EXPECT_TRUE(Remark.Hotness);
+ EXPECT_EQ(*Remark.Hotness, 4U);
+ EXPECT_EQ(Remark.Args.size(), 4U);
+
+ unsigned ArgID = 0;
+ for (const remarks::Argument &Arg : Remark.Args) {
+ switch (ArgID) {
+ case 0:
+ EXPECT_EQ(checkStr(Arg.Key, 6), "Callee");
+ EXPECT_EQ(checkStr(Arg.Val, 3), "bar");
+ EXPECT_FALSE(Arg.Loc);
+ break;
+ case 1:
+ EXPECT_EQ(checkStr(Arg.Key, 6), "String");
+ EXPECT_EQ(checkStr(Arg.Val, 26), " will not be inlined into ");
+ EXPECT_FALSE(Arg.Loc);
+ break;
+ case 2: {
+ EXPECT_EQ(checkStr(Arg.Key, 6), "Caller");
+ EXPECT_EQ(checkStr(Arg.Val, 3), "foo");
+ EXPECT_TRUE(Arg.Loc);
+ const remarks::RemarkLocation &RL = *Arg.Loc;
+ EXPECT_EQ(checkStr(RL.SourceFilePath, 6), "file.c");
+ EXPECT_EQ(RL.SourceLine, 2U);
+ EXPECT_EQ(RL.SourceColumn, 0U);
+ break;
+ }
+ case 3:
+ EXPECT_EQ(checkStr(Arg.Key, 6), "String");
+ EXPECT_EQ(checkStr(Arg.Val, 38),
+ " because its definition is unavailable");
+ EXPECT_FALSE(Arg.Loc);
+ break;
+ default:
+ break;
+ }
+ ++ArgID;
+ }
+
+ MaybeRemark = Parser.next();
+ Error E = MaybeRemark.takeError();
+ EXPECT_TRUE(E.isA<remarks::EndOfFileError>());
+ EXPECT_TRUE(errorToBool(std::move(E))); // Check for parsing errors.
+}
+
+static inline StringRef checkStr(LLVMRemarkStringRef Str,
+ unsigned ExpectedLen) {
+ const char *StrData = LLVMRemarkStringGetData(Str);
+ unsigned StrLen = LLVMRemarkStringGetLen(Str);
+ EXPECT_EQ(StrLen, ExpectedLen);
+ return StringRef(StrData, StrLen);
+}
+
+TEST(BitstreamRemarks, ContentsCAPI) {
+ remarks::StringTable BSStrTab;
+ remarks::Remark ToSerializeRemark;
+ ToSerializeRemark.RemarkType = remarks::Type::Missed;
+ ToSerializeRemark.PassName = "inline";
+ ToSerializeRemark.RemarkName = "NoDefinition";
+ ToSerializeRemark.FunctionName = "foo";
+ ToSerializeRemark.Loc = remarks::RemarkLocation{"file.c", 3, 12};
+ ToSerializeRemark.Hotness = 0;
+ ToSerializeRemark.Args.emplace_back();
+ ToSerializeRemark.Args.back().Key = "Callee";
+ ToSerializeRemark.Args.back().Val = "bar";
+ ToSerializeRemark.Args.emplace_back();
+ ToSerializeRemark.Args.back().Key = "String";
+ ToSerializeRemark.Args.back().Val = " will not be inlined into ";
+ ToSerializeRemark.Args.emplace_back();
+ ToSerializeRemark.Args.back().Key = "Caller";
+ ToSerializeRemark.Args.back().Val = "foo";
+ ToSerializeRemark.Args.back().Loc = remarks::RemarkLocation{"file.c", 2, 0};
+ ToSerializeRemark.Args.emplace_back();
+ ToSerializeRemark.Args.back().Key = "String";
+ ToSerializeRemark.Args.back().Val = " because its definition is unavailable";
+ BSStrTab.internalize(ToSerializeRemark);
+ std::string BSBuf;
+ raw_string_ostream BSStream(BSBuf);
+ Expected<std::unique_ptr<remarks::RemarkSerializer>> BSSerializer =
+ remarks::createRemarkSerializer(remarks::Format::Bitstream,
+ remarks::SerializerMode::Standalone,
+ BSStream, std::move(BSStrTab));
+ EXPECT_FALSE(errorToBool(BSSerializer.takeError()));
+ (*BSSerializer)->emit(ToSerializeRemark);
+
+ StringRef Buf = BSStream.str();
+ LLVMRemarkParserRef Parser =
+ LLVMRemarkParserCreateBitstream(Buf.data(), Buf.size());
+ LLVMRemarkEntryRef Remark = LLVMRemarkParserGetNext(Parser);
+ EXPECT_FALSE(Remark == nullptr);
+ EXPECT_EQ(LLVMRemarkEntryGetType(Remark), LLVMRemarkTypeMissed);
+ EXPECT_EQ(checkStr(LLVMRemarkEntryGetPassName(Remark), 6), "inline");
+ EXPECT_EQ(checkStr(LLVMRemarkEntryGetRemarkName(Remark), 12), "NoDefinition");
+ EXPECT_EQ(checkStr(LLVMRemarkEntryGetFunctionName(Remark), 3), "foo");
+ LLVMRemarkDebugLocRef DL = LLVMRemarkEntryGetDebugLoc(Remark);
+ EXPECT_EQ(checkStr(LLVMRemarkDebugLocGetSourceFilePath(DL), 6), "file.c");
+ EXPECT_EQ(LLVMRemarkDebugLocGetSourceLine(DL), 3U);
+ EXPECT_EQ(LLVMRemarkDebugLocGetSourceColumn(DL), 12U);
+ EXPECT_EQ(LLVMRemarkEntryGetHotness(Remark), 0U);
+ EXPECT_EQ(LLVMRemarkEntryGetNumArgs(Remark), 4U);
+
+ unsigned ArgID = 0;
+ LLVMRemarkArgRef Arg = LLVMRemarkEntryGetFirstArg(Remark);
+ do {
+ switch (ArgID) {
+ case 0:
+ EXPECT_EQ(checkStr(LLVMRemarkArgGetKey(Arg), 6), "Callee");
+ EXPECT_EQ(checkStr(LLVMRemarkArgGetValue(Arg), 3), "bar");
+ EXPECT_EQ(LLVMRemarkArgGetDebugLoc(Arg), nullptr);
+ break;
+ case 1:
+ EXPECT_EQ(checkStr(LLVMRemarkArgGetKey(Arg), 6), "String");
+ EXPECT_EQ(checkStr(LLVMRemarkArgGetValue(Arg), 26),
+ " will not be inlined into ");
+ EXPECT_EQ(LLVMRemarkArgGetDebugLoc(Arg), nullptr);
+ break;
+ case 2: {
+ EXPECT_EQ(checkStr(LLVMRemarkArgGetKey(Arg), 6), "Caller");
+ EXPECT_EQ(checkStr(LLVMRemarkArgGetValue(Arg), 3), "foo");
+ LLVMRemarkDebugLocRef DL = LLVMRemarkArgGetDebugLoc(Arg);
+ EXPECT_EQ(checkStr(LLVMRemarkDebugLocGetSourceFilePath(DL), 6), "file.c");
+ EXPECT_EQ(LLVMRemarkDebugLocGetSourceLine(DL), 2U);
+ EXPECT_EQ(LLVMRemarkDebugLocGetSourceColumn(DL), 0U);
+ break;
+ }
+ case 3:
+ EXPECT_EQ(checkStr(LLVMRemarkArgGetKey(Arg), 6), "String");
+ EXPECT_EQ(checkStr(LLVMRemarkArgGetValue(Arg), 38),
+ " because its definition is unavailable");
+ EXPECT_EQ(LLVMRemarkArgGetDebugLoc(Arg), nullptr);
+ break;
+ default:
+ break;
+ }
+ ++ArgID;
+ } while ((Arg = LLVMRemarkEntryGetNextArg(Arg, Remark)));
+
+ LLVMRemarkEntryDispose(Remark);
+
+ EXPECT_EQ(LLVMRemarkParserGetNext(Parser), nullptr);
+
+ EXPECT_FALSE(LLVMRemarkParserHasError(Parser));
+ LLVMRemarkParserDispose(Parser);
+}
+
+static void parseBad(StringRef Input, const char *ErrorMsg) {
+ Expected<std::unique_ptr<remarks::RemarkParser>> MaybeBSParser =
+ remarks::createRemarkParser(remarks::Format::Bitstream, Input);
+ EXPECT_FALSE(errorToBool(MaybeBSParser.takeError()));
+ EXPECT_TRUE(*MaybeBSParser != nullptr);
+
+ remarks::RemarkParser &BSParser = **MaybeBSParser;
+ Expected<std::unique_ptr<remarks::Remark>> BSRemark = BSParser.next();
+ EXPECT_EQ(ErrorMsg, toString(BSRemark.takeError())); // Expect an error.
+}
+
+TEST(BitstreamRemarks, ParsingEmpty) {
+ parseBad(StringRef(), "End of file reached.");
+}
+
+TEST(BitstreamRemarks, ParsingBadMagic) {
+ parseBad("KRMR", "Unknown magic number: expecting RMRK, got KRMR.");
+}
+
+// Testing malformed bitstream is not easy. We would need to replace bytes in
+// the stream to create malformed and unknown records and blocks. There is no
+// textual format for bitstream that can be decoded, modified and encoded
+// back.
+
+// FIXME: Add tests for the following error messages:
+// * Error while parsing META_BLOCK: malformed record entry
+// (RECORD_META_CONTAINER_INFO).
+// * Error while parsing META_BLOCK: malformed record entry
+// (RECORD_META_REMARK_VERSION).
+// * Error while parsing META_BLOCK: malformed record entry
+// (RECORD_META_STRTAB).
+// * Error while parsing META_BLOCK: malformed record entry
+// (RECORD_META_EXTERNAL_FILE).
+// * Error while parsing META_BLOCK: unknown record entry (NUM).
+// * Error while parsing REMARK_BLOCK: malformed record entry
+// (RECORD_REMARK_HEADER).
+// * Error while parsing REMARK_BLOCK: malformed record entry
+// (RECORD_REMARK_DEBUG_LOC).
+// * Error while parsing REMARK_BLOCK: malformed record entry
+// (RECORD_REMARK_HOTNESS).
+// * Error while parsing REMARK_BLOCK: malformed record entry
+// (RECORD_REMARK_ARG_WITH_DEBUGLOC).
+// * Error while parsing REMARK_BLOCK: malformed record entry
+// (RECORD_REMARK_ARG_WITHOUT_DEBUGLOC).
+// * Error while parsing REMARK_BLOCK: unknown record entry (NUM).
+// * Error while parsing META_BLOCK: expecting [ENTER_SUBBLOCO, META_BLOCK,
+// ...].
+// * Error while entering META_BLOCK.
+// * Error while parsing META_BLOCK: expecting records.
+// * Error while parsing META_BLOCK: unterminated block.
+// * Error while parsing REMARK_BLOCK: expecting [ENTER_SUBBLOCO, REMARK_BLOCK,
+// ...].
+// * Error while entering REMARK_BLOCK.
+// * Error while parsing REMARK_BLOCK: expecting records.
+// * Error while parsing REMARK_BLOCK: unterminated block.
+// * Error while parsing BLOCKINFO_BLOCK: expecting [ENTER_SUBBLOCK,
+// BLOCKINFO_BLOCK, ...].
+// * Error while parsing BLOCKINFO_BLOCK.
+// * Unexpected error while parsing bitstream.
+// * Expecting META_BLOCK after the BLOCKINFO_BLOCK.
+// * Error while parsing BLOCK_META: missing container version.
+// * Error while parsing BLOCK_META: invalid container type.
+// * Error while parsing BLOCK_META: missing container type.
+// * Error while parsing BLOCK_META: missing string table.
+// * Error while parsing BLOCK_META: missing remark version.
+// * Error while parsing BLOCK_META: missing external file path.
+// * Error while parsing external file's BLOCK_META: wrong container type.
+// * Error while parsing external file's BLOCK_META: mismatching versions:
+// original meta: NUM, external file meta: NUM.
+// * Error while parsing BLOCK_REMARK: missing string table.
+// * Error while parsing BLOCK_REMARK: missing remark type.
+// * Error while parsing BLOCK_REMARK: unknown remark type.
+// * Error while parsing BLOCK_REMARK: missing remark name.
+// * Error while parsing BLOCK_REMARK: missing remark pass.
+// * Error while parsing BLOCK_REMARK: missing remark function name.
+// * Error while parsing BLOCK_REMARK: missing key in remark argument.
+// * Error while parsing BLOCK_REMARK: missing value in remark argument.
Modified: llvm/trunk/unittests/Remarks/CMakeLists.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/unittests/Remarks/CMakeLists.txt?rev=371429&r1=371428&r2=371429&view=diff
==============================================================================
--- llvm/trunk/unittests/Remarks/CMakeLists.txt (original)
+++ llvm/trunk/unittests/Remarks/CMakeLists.txt Mon Sep 9 10:43:50 2019
@@ -6,6 +6,7 @@ set(LLVM_LINK_COMPONENTS
add_llvm_unittest(RemarksTests
BitstreamRemarksFormatTest.cpp
+ BitstreamRemarksParsingTest.cpp
BitstreamRemarksSerializerTest.cpp
RemarksAPITest.cpp
RemarksStrTabParsingTest.cpp
More information about the llvm-commits
mailing list