[llvm] r342283 - [SampleFDO] Add FunctionOffsetTable in compact binary format profile.
Alexander Kornienko via llvm-commits
llvm-commits at lists.llvm.org
Mon Sep 17 03:38:15 PDT 2018
Another problem with this test is that it tries to write a file in the
current directory, which may be read-only in some setups. Is there a reason
to access the real file system in this test? Would you mind me switching
this to in-memory streams?
On Sat, Sep 15, 2018 at 2:06 AM Wei Mi via llvm-commits <
llvm-commits at lists.llvm.org> wrote:
> Should be fixed in rL342301.
>
> Thanks!
> Wei.
>
> On Fri, Sep 14, 2018 at 4:35 PM, Wei Mi <wmi at google.com> wrote:
>
>> There is file system race indeed. I will fix that. Thanks Reid!
>>
>> Wei.
>>
>> On Fri, Sep 14, 2018 at 4:09 PM, Reid Kleckner <rnk at google.com> wrote:
>>
>>> These tests seem flaky when I run them locally. Are there possible
>>> filesystem race issues here?
>>>
>>> FAIL: LLVM-Unit ::
>>> ProfileData/./ProfileDataTests.exe/SampleProfTest.roundtrip_compact_binary_profile
>>> (2329 of 27766)
>>> ******************** TEST 'LLVM-Unit ::
>>> ProfileData/./ProfileDataTests.exe/SampleProfTest.roundtrip_compact_binary_profile'
>>> FAILED ********************
>>> Note: Google Test filter =
>>> SampleProfTest.roundtrip_compact_binary_profile
>>> [==========] Running 1 test from 1 test case.
>>> [----------] Global test environment set-up.
>>> [----------] 1 test from SampleProfTest
>>> [ RUN ] SampleProfTest.roundtrip_compact_binary_profile
>>> C:\src\llvm-project\llvm\unittests\ProfileData\SampleProfTest.cpp(59):
>>> error: Value of: NoError(ReaderOrErr.getError())
>>> Actual: false (error 6: Unrecognized sample profile encoding format)
>>> Expected: true
>>> unknown file: error: SEH exception with code 0x3221225477 thrown in the
>>> test body.
>>> unknown file: error: SEH exception with code 0x3221225477 thrown in the
>>> test fixture's destructor.
>>> [ FAILED ] SampleProfTest.roundtrip_compact_binary_profile (2 ms)
>>> [----------] 1 test from SampleProfTest (2 ms total)
>>>
>>> [----------] Global test environment tear-down
>>> [==========] 1 test from 1 test case ran. (2 ms total)
>>> [ PASSED ] 0 tests.
>>> [ FAILED ] 1 test, listed below:
>>> [ FAILED ] SampleProfTest.roundtrip_compact_binary_profile
>>>
>>> 1 FAILED TEST
>>>
>>> ********************
>>> Testing: 0 ..
>>> FAIL: LLVM-Unit ::
>>> ProfileData/./ProfileDataTests.exe/SampleProfTest.roundtrip_text_profile
>>> (2330 of 27766)
>>> ******************** TEST 'LLVM-Unit ::
>>> ProfileData/./ProfileDataTests.exe/SampleProfTest.roundtrip_text_profile'
>>> FAILED ********************
>>> Note: Google Test filter = SampleProfTest.roundtrip_text_profile
>>> [==========] Running 1 test from 1 test case.
>>> [----------] Global test environment set-up.
>>> [----------] 1 test from SampleProfTest
>>> [ RUN ] SampleProfTest.roundtrip_text_profile
>>> C:\src\llvm-project\llvm\unittests\ProfileData\SampleProfTest.cpp(59):
>>> error: Value of: NoError(ReaderOrErr.getError())
>>> Actual: false (error 6: Unrecognized sample profile encoding format)
>>> Expected: true
>>> unknown file: error: SEH exception with code 0x3221225477 thrown in the
>>> test body.
>>> unknown file: error: SEH exception with code 0x3221225477 thrown in the
>>> test fixture's destructor.
>>> [ FAILED ] SampleProfTest.roundtrip_text_profile (2 ms)
>>> [----------] 1 test from SampleProfTest (2 ms total)
>>>
>>> [----------] Global test environment tear-down
>>> [==========] 1 test from 1 test case ran. (2 ms total)
>>> [ PASSED ] 0 tests.
>>> [ FAILED ] 1 test, listed below:
>>> [ FAILED ] SampleProfTest.roundtrip_text_profile
>>>
>>> 1 FAILED TEST
>>>
>>> ********************
>>>
>>> On Fri, Sep 14, 2018 at 1:54 PM Wei Mi via llvm-commits <
>>> llvm-commits at lists.llvm.org> wrote:
>>>
>>>> Author: wmi
>>>> Date: Fri Sep 14 13:52:59 2018
>>>> New Revision: 342283
>>>>
>>>> URL: http://llvm.org/viewvc/llvm-project?rev=342283&view=rev
>>>> Log:
>>>> [SampleFDO] Add FunctionOffsetTable in compact binary format profile.
>>>>
>>>> The patch saves a function offset table which maps function name index
>>>> to the
>>>> offset of its function profile to the start of the binary profile. By
>>>> using
>>>> the function offset table, for those function profiles which will not
>>>> be used
>>>> when compiling a module, the profile reader does't have to read them.
>>>> For
>>>> profile size around 10~20M, it saves ~10% compile time.
>>>>
>>>> Differential Revision: https://reviews.llvm.org/D51863
>>>>
>>>>
>>>> Modified:
>>>> llvm/trunk/include/llvm/ProfileData/SampleProf.h
>>>> llvm/trunk/include/llvm/ProfileData/SampleProfReader.h
>>>> llvm/trunk/include/llvm/ProfileData/SampleProfWriter.h
>>>> llvm/trunk/lib/ProfileData/SampleProf.cpp
>>>> llvm/trunk/lib/ProfileData/SampleProfReader.cpp
>>>> llvm/trunk/lib/ProfileData/SampleProfWriter.cpp
>>>> llvm/trunk/lib/Transforms/IPO/SampleProfile.cpp
>>>>
>>>> llvm/trunk/test/Transforms/SampleProfile/Inputs/function_metadata.compact.afdo
>>>>
>>>> llvm/trunk/test/Transforms/SampleProfile/Inputs/indirect-call.compact.afdo
>>>>
>>>> llvm/trunk/test/Transforms/SampleProfile/Inputs/inline.compactbinary.afdo
>>>> llvm/trunk/unittests/ProfileData/SampleProfTest.cpp
>>>>
>>>> Modified: llvm/trunk/include/llvm/ProfileData/SampleProf.h
>>>> URL:
>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/ProfileData/SampleProf.h?rev=342283&r1=342282&r2=342283&view=diff
>>>>
>>>> ==============================================================================
>>>> --- llvm/trunk/include/llvm/ProfileData/SampleProf.h (original)
>>>> +++ llvm/trunk/include/llvm/ProfileData/SampleProf.h Fri Sep 14
>>>> 13:52:59 2018
>>>> @@ -49,7 +49,8 @@ enum class sampleprof_error {
>>>> unsupported_writing_format,
>>>> truncated_name_table,
>>>> not_implemented,
>>>> - counter_overflow
>>>> + counter_overflow,
>>>> + ostream_seek_unsupported
>>>> };
>>>>
>>>> inline std::error_code make_error_code(sampleprof_error E) {
>>>>
>>>> Modified: llvm/trunk/include/llvm/ProfileData/SampleProfReader.h
>>>> URL:
>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/ProfileData/SampleProfReader.h?rev=342283&r1=342282&r2=342283&view=diff
>>>>
>>>> ==============================================================================
>>>> --- llvm/trunk/include/llvm/ProfileData/SampleProfReader.h (original)
>>>> +++ llvm/trunk/include/llvm/ProfileData/SampleProfReader.h Fri Sep 14
>>>> 13:52:59 2018
>>>> @@ -279,6 +279,8 @@ public:
>>>> /// Print the profile for \p FName on stream \p OS.
>>>> void dumpFunctionProfile(StringRef FName, raw_ostream &OS = dbgs());
>>>>
>>>> + virtual void collectFuncsToUse(const Module &M) {}
>>>> +
>>>> /// Print all the profiles on stream \p OS.
>>>> void dump(raw_ostream &OS = dbgs());
>>>>
>>>> @@ -364,7 +366,7 @@ public:
>>>> : SampleProfileReader(std::move(B), C, Format) {}
>>>>
>>>> /// Read and validate the file header.
>>>> - std::error_code readHeader() override;
>>>> + virtual std::error_code readHeader() override;
>>>>
>>>> /// Read sample profiles from the associated file.
>>>> std::error_code read() override;
>>>> @@ -378,6 +380,10 @@ protected:
>>>> /// \returns the read value.
>>>> template <typename T> ErrorOr<T> readNumber();
>>>>
>>>> + /// Read a numeric value of type T from the profile. The value is
>>>> saved
>>>> + /// without encoded.
>>>> + template <typename T> ErrorOr<T> readUnencodedNumber();
>>>> +
>>>> /// Read a string from the profile.
>>>> ///
>>>> /// If an error occurs during decoding, a diagnostic message is
>>>> emitted and
>>>> @@ -392,6 +398,9 @@ protected:
>>>> /// Return true if we've reached the end of file.
>>>> bool at_eof() const { return Data >= End; }
>>>>
>>>> + /// Read the next function profile instance.
>>>> + std::error_code readFuncProfile();
>>>> +
>>>> /// Read the contents of the given profile instance.
>>>> std::error_code readProfile(FunctionSamples &FProfile);
>>>>
>>>> @@ -436,10 +445,17 @@ class SampleProfileReaderCompactBinary :
>>>> private:
>>>> /// Function name table.
>>>> std::vector<std::string> NameTable;
>>>> + /// The table mapping from function name to the offset of its
>>>> FunctionSample
>>>> + /// towards file start.
>>>> + DenseMap<StringRef, uint64_t> FuncOffsetTable;
>>>> + /// The set containing the functions to use when compiling a module.
>>>> + DenseSet<StringRef> FuncsToUse;
>>>> virtual std::error_code verifySPMagic(uint64_t Magic) override;
>>>> virtual std::error_code readNameTable() override;
>>>> /// Read a string indirectly via the name table.
>>>> virtual ErrorOr<StringRef> readStringFromTable() override;
>>>> + virtual std::error_code readHeader() override;
>>>> + std::error_code readFuncOffsetTable();
>>>>
>>>> public:
>>>> SampleProfileReaderCompactBinary(std::unique_ptr<MemoryBuffer> B,
>>>> @@ -448,6 +464,12 @@ public:
>>>>
>>>> /// \brief Return true if \p Buffer is in the format supported by
>>>> this class.
>>>> static bool hasFormat(const MemoryBuffer &Buffer);
>>>> +
>>>> + /// Read samples only for functions to use.
>>>> + std::error_code read() override;
>>>> +
>>>> + /// Collect functions to be used when compiling Module \p M.
>>>> + void collectFuncsToUse(const Module &M) override;
>>>> };
>>>>
>>>> using InlineCallStack = SmallVector<FunctionSamples *, 10>;
>>>>
>>>> Modified: llvm/trunk/include/llvm/ProfileData/SampleProfWriter.h
>>>> URL:
>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/ProfileData/SampleProfWriter.h?rev=342283&r1=342282&r2=342283&view=diff
>>>>
>>>> ==============================================================================
>>>> --- llvm/trunk/include/llvm/ProfileData/SampleProfWriter.h (original)
>>>> +++ llvm/trunk/include/llvm/ProfileData/SampleProfWriter.h Fri Sep 14
>>>> 13:52:59 2018
>>>> @@ -42,7 +42,7 @@ public:
>>>> /// Write all the sample profiles in the given map of samples.
>>>> ///
>>>> /// \returns status code of the file update operation.
>>>> - std::error_code write(const StringMap<FunctionSamples> &ProfileMap);
>>>> + virtual std::error_code write(const StringMap<FunctionSamples>
>>>> &ProfileMap);
>>>>
>>>> raw_ostream &getOutputStream() { return *OutputStream; }
>>>>
>>>> @@ -103,14 +103,15 @@ private:
>>>> /// Sample-based profile writer (binary format).
>>>> class SampleProfileWriterBinary : public SampleProfileWriter {
>>>> public:
>>>> - std::error_code write(const FunctionSamples &S) override;
>>>> + virtual std::error_code write(const FunctionSamples &S) override;
>>>> SampleProfileWriterBinary(std::unique_ptr<raw_ostream> &OS)
>>>> : SampleProfileWriter(OS) {}
>>>>
>>>> protected:
>>>> virtual std::error_code writeNameTable() = 0;
>>>> virtual std::error_code writeMagicIdent() = 0;
>>>> - std::error_code writeHeader(const StringMap<FunctionSamples>
>>>> &ProfileMap) override;
>>>> + virtual std::error_code
>>>> + writeHeader(const StringMap<FunctionSamples> &ProfileMap) override;
>>>> std::error_code writeSummary();
>>>> std::error_code writeNameIdx(StringRef FName);
>>>> std::error_code writeBody(const FunctionSamples &S);
>>>> @@ -135,12 +136,56 @@ protected:
>>>> virtual std::error_code writeMagicIdent() override;
>>>> };
>>>>
>>>> +// CompactBinary is a compact format of binary profile which both
>>>> reduces
>>>> +// the profile size and the load time needed when compiling. It has two
>>>> +// major difference with Binary format.
>>>> +// 1. It represents all the strings in name table using md5 hash.
>>>> +// 2. It saves a function offset table which maps function name index
>>>> to
>>>> +// the offset of its function profile to the start of the binary
>>>> profile,
>>>> +// so by using the function offset table, for those function profiles
>>>> which
>>>> +// will not be needed when compiling a module, the profile reader
>>>> does't
>>>> +// have to read them and it saves compile time if the profile size is
>>>> huge.
>>>> +// The layout of the compact format is shown as follows:
>>>> +//
>>>> +// Part1: Profile header, the same as binary format, containing
>>>> magic
>>>> +// number, version, summary, name table...
>>>> +// Part2: Function Offset Table Offset, which saves the position of
>>>> +// Part4.
>>>> +// Part3: Function profile collection
>>>> +// function1 profile start
>>>> +// ....
>>>> +// function2 profile start
>>>> +// ....
>>>> +// function3 profile start
>>>> +// ....
>>>> +// ......
>>>> +// Part4: Function Offset Table
>>>> +// function1 name index --> function1 profile start
>>>> +// function2 name index --> function2 profile start
>>>> +// function3 name index --> function3 profile start
>>>> +//
>>>> +// We need Part2 because profile reader can use it to find out and read
>>>> +// function offset table without reading Part3 first.
>>>> class SampleProfileWriterCompactBinary : public
>>>> SampleProfileWriterBinary {
>>>> using SampleProfileWriterBinary::SampleProfileWriterBinary;
>>>>
>>>> +public:
>>>> + virtual std::error_code write(const FunctionSamples &S) override;
>>>> + virtual std::error_code
>>>> + write(const StringMap<FunctionSamples> &ProfileMap) override;
>>>> +
>>>> protected:
>>>> + /// The table mapping from function name to the offset of its
>>>> FunctionSample
>>>> + /// towards profile start.
>>>> + MapVector<StringRef, uint64_t> FuncOffsetTable;
>>>> + /// The offset of the slot to be filled with the offset of
>>>> FuncOffsetTable
>>>> + /// towards profile start.
>>>> + uint64_t TableOffset;
>>>> virtual std::error_code writeNameTable() override;
>>>> virtual std::error_code writeMagicIdent() override;
>>>> + virtual std::error_code
>>>> + writeHeader(const StringMap<FunctionSamples> &ProfileMap) override;
>>>> + std::error_code writeFuncOffsetTable();
>>>> };
>>>>
>>>> } // end namespace sampleprof
>>>>
>>>> Modified: llvm/trunk/lib/ProfileData/SampleProf.cpp
>>>> URL:
>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/ProfileData/SampleProf.cpp?rev=342283&r1=342282&r2=342283&view=diff
>>>>
>>>> ==============================================================================
>>>> --- llvm/trunk/lib/ProfileData/SampleProf.cpp (original)
>>>> +++ llvm/trunk/lib/ProfileData/SampleProf.cpp Fri Sep 14 13:52:59 2018
>>>> @@ -67,6 +67,8 @@ class SampleProfErrorCategoryType : publ
>>>> return "Unimplemented feature";
>>>> case sampleprof_error::counter_overflow:
>>>> return "Counter overflow";
>>>> + case sampleprof_error::ostream_seek_unsupported:
>>>> + return "Ostream does not support seek";
>>>> }
>>>> llvm_unreachable("A value of sampleprof_error has no message.");
>>>> }
>>>>
>>>> Modified: llvm/trunk/lib/ProfileData/SampleProfReader.cpp
>>>> URL:
>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/ProfileData/SampleProfReader.cpp?rev=342283&r1=342282&r2=342283&view=diff
>>>>
>>>> ==============================================================================
>>>> --- llvm/trunk/lib/ProfileData/SampleProfReader.cpp (original)
>>>> +++ llvm/trunk/lib/ProfileData/SampleProfReader.cpp Fri Sep 14 13:52:59
>>>> 2018
>>>> @@ -30,6 +30,7 @@
>>>> #include "llvm/Support/ErrorOr.h"
>>>> #include "llvm/Support/LEB128.h"
>>>> #include "llvm/Support/LineIterator.h"
>>>> +#include "llvm/Support/MD5.h"
>>>> #include "llvm/Support/MemoryBuffer.h"
>>>> #include "llvm/Support/raw_ostream.h"
>>>> #include <algorithm>
>>>> @@ -320,6 +321,21 @@ ErrorOr<StringRef> SampleProfileReaderBi
>>>> }
>>>>
>>>> template <typename T>
>>>> +ErrorOr<T> SampleProfileReaderBinary::readUnencodedNumber() {
>>>> + std::error_code EC;
>>>> +
>>>> + if (Data + sizeof(T) > End) {
>>>> + EC = sampleprof_error::truncated;
>>>> + reportError(0, EC.message());
>>>> + return EC;
>>>> + }
>>>> +
>>>> + using namespace support;
>>>> + T Val = endian::readNext<T, little, unaligned>(Data);
>>>> + return Val;
>>>> +}
>>>> +
>>>> +template <typename T>
>>>> inline ErrorOr<uint32_t> SampleProfileReaderBinary::readStringIndex(T
>>>> &Table) {
>>>> std::error_code EC;
>>>> auto Idx = readNumber<uint32_t>();
>>>> @@ -423,29 +439,51 @@ SampleProfileReaderBinary::readProfile(F
>>>> return sampleprof_error::success;
>>>> }
>>>>
>>>> -std::error_code SampleProfileReaderBinary::read() {
>>>> - while (!at_eof()) {
>>>> - auto NumHeadSamples = readNumber<uint64_t>();
>>>> - if (std::error_code EC = NumHeadSamples.getError())
>>>> - return EC;
>>>> +std::error_code SampleProfileReaderBinary::readFuncProfile() {
>>>> + auto NumHeadSamples = readNumber<uint64_t>();
>>>> + if (std::error_code EC = NumHeadSamples.getError())
>>>> + return EC;
>>>>
>>>> - auto FName(readStringFromTable());
>>>> - if (std::error_code EC = FName.getError())
>>>> - return EC;
>>>> + auto FName(readStringFromTable());
>>>> + if (std::error_code EC = FName.getError())
>>>> + return EC;
>>>>
>>>> - Profiles[*FName] = FunctionSamples();
>>>> - FunctionSamples &FProfile = Profiles[*FName];
>>>> - FProfile.setName(*FName);
>>>> + Profiles[*FName] = FunctionSamples();
>>>> + FunctionSamples &FProfile = Profiles[*FName];
>>>> + FProfile.setName(*FName);
>>>>
>>>> - FProfile.addHeadSamples(*NumHeadSamples);
>>>> + FProfile.addHeadSamples(*NumHeadSamples);
>>>> +
>>>> + if (std::error_code EC = readProfile(FProfile))
>>>> + return EC;
>>>> + return sampleprof_error::success;
>>>> +}
>>>>
>>>> - if (std::error_code EC = readProfile(FProfile))
>>>> +std::error_code SampleProfileReaderBinary::read() {
>>>> + while (!at_eof()) {
>>>> + if (std::error_code EC = readFuncProfile())
>>>> return EC;
>>>> }
>>>>
>>>> return sampleprof_error::success;
>>>> }
>>>>
>>>> +std::error_code SampleProfileReaderCompactBinary::read() {
>>>> + for (auto Name : FuncsToUse) {
>>>> + auto GUID = std::to_string(MD5Hash(Name));
>>>> + auto iter = FuncOffsetTable.find(StringRef(GUID));
>>>> + if (iter == FuncOffsetTable.end())
>>>> + continue;
>>>> + const uint8_t *SavedData = Data;
>>>> + Data = reinterpret_cast<const uint8_t *>(Buffer->getBufferStart())
>>>> +
>>>> + iter->second;
>>>> + if (std::error_code EC = readFuncProfile())
>>>> + return EC;
>>>> + Data = SavedData;
>>>> + }
>>>> + return sampleprof_error::success;
>>>> +}
>>>> +
>>>> std::error_code SampleProfileReaderRawBinary::verifySPMagic(uint64_t
>>>> Magic) {
>>>> if (Magic == SPMagic())
>>>> return sampleprof_error::success;
>>>> @@ -514,6 +552,53 @@ std::error_code SampleProfileReaderBinar
>>>> return sampleprof_error::success;
>>>> }
>>>>
>>>> +std::error_code SampleProfileReaderCompactBinary::readHeader() {
>>>> + SampleProfileReaderBinary::readHeader();
>>>> + if (std::error_code EC = readFuncOffsetTable())
>>>> + return EC;
>>>> + return sampleprof_error::success;
>>>> +}
>>>> +
>>>> +std::error_code
>>>> SampleProfileReaderCompactBinary::readFuncOffsetTable() {
>>>> + auto TableOffset = readUnencodedNumber<uint64_t>();
>>>> + if (std::error_code EC = TableOffset.getError())
>>>> + return EC;
>>>> +
>>>> + const uint8_t *SavedData = Data;
>>>> + const uint8_t *TableStart =
>>>> + reinterpret_cast<const uint8_t *>(Buffer->getBufferStart()) +
>>>> + *TableOffset;
>>>> + Data = TableStart;
>>>> +
>>>> + auto Size = readNumber<uint64_t>();
>>>> + if (std::error_code EC = Size.getError())
>>>> + return EC;
>>>> +
>>>> + FuncOffsetTable.reserve(*Size);
>>>> + for (uint32_t I = 0; I < *Size; ++I) {
>>>> + auto FName(readStringFromTable());
>>>> + if (std::error_code EC = FName.getError())
>>>> + return EC;
>>>> +
>>>> + auto Offset = readNumber<uint64_t>();
>>>> + if (std::error_code EC = Offset.getError())
>>>> + return EC;
>>>> +
>>>> + FuncOffsetTable[*FName] = *Offset;
>>>> + }
>>>> + End = TableStart;
>>>> + Data = SavedData;
>>>> + return sampleprof_error::success;
>>>> +}
>>>> +
>>>> +void SampleProfileReaderCompactBinary::collectFuncsToUse(const Module
>>>> &M) {
>>>> + FuncsToUse.clear();
>>>> + for (auto &F : M) {
>>>> + StringRef Fname = F.getName().split('.').first;
>>>> + FuncsToUse.insert(Fname);
>>>> + }
>>>> +}
>>>> +
>>>> std::error_code SampleProfileReaderBinary::readSummaryEntry(
>>>> std::vector<ProfileSummaryEntry> &Entries) {
>>>> auto Cutoff = readNumber<uint64_t>();
>>>>
>>>> Modified: llvm/trunk/lib/ProfileData/SampleProfWriter.cpp
>>>> URL:
>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/ProfileData/SampleProfWriter.cpp?rev=342283&r1=342282&r2=342283&view=diff
>>>>
>>>> ==============================================================================
>>>> --- llvm/trunk/lib/ProfileData/SampleProfWriter.cpp (original)
>>>> +++ llvm/trunk/lib/ProfileData/SampleProfWriter.cpp Fri Sep 14 13:52:59
>>>> 2018
>>>> @@ -22,6 +22,8 @@
>>>> #include "llvm/ADT/StringRef.h"
>>>> #include "llvm/ProfileData/ProfileCommon.h"
>>>> #include "llvm/ProfileData/SampleProf.h"
>>>> +#include "llvm/Support/Endian.h"
>>>> +#include "llvm/Support/EndianStream.h"
>>>> #include "llvm/Support/ErrorOr.h"
>>>> #include "llvm/Support/FileSystem.h"
>>>> #include "llvm/Support/LEB128.h"
>>>> @@ -64,6 +66,15 @@ SampleProfileWriter::write(const StringM
>>>> return sampleprof_error::success;
>>>> }
>>>>
>>>> +std::error_code SampleProfileWriterCompactBinary::write(
>>>> + const StringMap<FunctionSamples> &ProfileMap) {
>>>> + if (std::error_code EC = SampleProfileWriter::write(ProfileMap))
>>>> + return EC;
>>>> + if (std::error_code EC = writeFuncOffsetTable())
>>>> + return EC;
>>>> + return sampleprof_error::success;
>>>> +}
>>>> +
>>>> /// Write samples to a text file.
>>>> ///
>>>> /// Note: it may be tempting to implement this in terms of
>>>> @@ -168,6 +179,30 @@ std::error_code SampleProfileWriterRawBi
>>>> return sampleprof_error::success;
>>>> }
>>>>
>>>> +std::error_code
>>>> SampleProfileWriterCompactBinary::writeFuncOffsetTable() {
>>>> + auto &OS = *OutputStream;
>>>> +
>>>> + // Fill the slot remembered by TableOffset with the offset of
>>>> FuncOffsetTable.
>>>> + auto &OFS = static_cast<raw_fd_ostream &>(OS);
>>>> + uint64_t FuncOffsetTableStart = OS.tell();
>>>> + if (OFS.seek(TableOffset) == (uint64_t)-1)
>>>> + return sampleprof_error::ostream_seek_unsupported;
>>>> + support::endian::Writer Writer(*OutputStream, support::little);
>>>> + Writer.write(FuncOffsetTableStart);
>>>> + if (OFS.seek(FuncOffsetTableStart) == (uint64_t)-1)
>>>> + return sampleprof_error::ostream_seek_unsupported;
>>>> +
>>>> + // Write out the table size.
>>>> + encodeULEB128(FuncOffsetTable.size(), OS);
>>>> +
>>>> + // Write out FuncOffsetTable.
>>>> + for (auto entry : FuncOffsetTable) {
>>>> + writeNameIdx(entry.first);
>>>> + encodeULEB128(entry.second, OS);
>>>> + }
>>>> + return sampleprof_error::success;
>>>> +}
>>>> +
>>>> std::error_code SampleProfileWriterCompactBinary::writeNameTable() {
>>>> auto &OS = *OutputStream;
>>>> std::set<StringRef> V;
>>>> @@ -215,6 +250,19 @@ std::error_code SampleProfileWriterBinar
>>>> return sampleprof_error::success;
>>>> }
>>>>
>>>> +std::error_code SampleProfileWriterCompactBinary::writeHeader(
>>>> + const StringMap<FunctionSamples> &ProfileMap) {
>>>> + support::endian::Writer Writer(*OutputStream, support::little);
>>>> + if (auto EC = SampleProfileWriterBinary::writeHeader(ProfileMap))
>>>> + return EC;
>>>> +
>>>> + // Reserve a slot for the offset of function offset table. The slot
>>>> will
>>>> + // be populated with the offset of FuncOffsetTable later.
>>>> + TableOffset = OutputStream->tell();
>>>> + Writer.write(static_cast<uint64_t>(-2));
>>>> + return sampleprof_error::success;
>>>> +}
>>>> +
>>>> std::error_code SampleProfileWriterBinary::writeSummary() {
>>>> auto &OS = *OutputStream;
>>>> encodeULEB128(Summary->getTotalCount(), OS);
>>>> @@ -282,6 +330,15 @@ std::error_code SampleProfileWriterBinar
>>>> encodeULEB128(S.getHeadSamples(), *OutputStream);
>>>> return writeBody(S);
>>>> }
>>>> +
>>>> +std::error_code
>>>> +SampleProfileWriterCompactBinary::write(const FunctionSamples &S) {
>>>> + uint64_t Offset = OutputStream->tell();
>>>> + StringRef Name = S.getName();
>>>> + FuncOffsetTable[Name] = Offset;
>>>> + encodeULEB128(S.getHeadSamples(), *OutputStream);
>>>> + return writeBody(S);
>>>> +}
>>>>
>>>> /// Create a sample profile file writer based on the specified format.
>>>> ///
>>>>
>>>> Modified: llvm/trunk/lib/Transforms/IPO/SampleProfile.cpp
>>>> URL:
>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/IPO/SampleProfile.cpp?rev=342283&r1=342282&r2=342283&view=diff
>>>>
>>>> ==============================================================================
>>>> --- llvm/trunk/lib/Transforms/IPO/SampleProfile.cpp (original)
>>>> +++ llvm/trunk/lib/Transforms/IPO/SampleProfile.cpp Fri Sep 14 13:52:59
>>>> 2018
>>>> @@ -1515,6 +1515,7 @@ bool SampleProfileLoader::doInitializati
>>>> return false;
>>>> }
>>>> Reader = std::move(ReaderOrErr.get());
>>>> + Reader->collectFuncsToUse(M);
>>>> ProfileIsValid = (Reader->read() == sampleprof_error::success);
>>>> return true;
>>>> }
>>>>
>>>> Modified:
>>>> llvm/trunk/test/Transforms/SampleProfile/Inputs/function_metadata.compact.afdo
>>>> URL:
>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SampleProfile/Inputs/function_metadata.compact.afdo?rev=342283&r1=342282&r2=342283&view=diff
>>>>
>>>> ==============================================================================
>>>> Binary files - no diff available.
>>>>
>>>> Modified:
>>>> llvm/trunk/test/Transforms/SampleProfile/Inputs/indirect-call.compact.afdo
>>>> URL:
>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SampleProfile/Inputs/indirect-call.compact.afdo?rev=342283&r1=342282&r2=342283&view=diff
>>>>
>>>> ==============================================================================
>>>> Binary files - no diff available.
>>>>
>>>> Modified:
>>>> llvm/trunk/test/Transforms/SampleProfile/Inputs/inline.compactbinary.afdo
>>>> URL:
>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SampleProfile/Inputs/inline.compactbinary.afdo?rev=342283&r1=342282&r2=342283&view=diff
>>>>
>>>> ==============================================================================
>>>> Binary files - no diff available.
>>>>
>>>> Modified: llvm/trunk/unittests/ProfileData/SampleProfTest.cpp
>>>> URL:
>>>> http://llvm.org/viewvc/llvm-project/llvm/trunk/unittests/ProfileData/SampleProfTest.cpp?rev=342283&r1=342282&r2=342283&view=diff
>>>>
>>>> ==============================================================================
>>>> --- llvm/trunk/unittests/ProfileData/SampleProfTest.cpp (original)
>>>> +++ llvm/trunk/unittests/ProfileData/SampleProfTest.cpp Fri Sep 14
>>>> 13:52:59 2018
>>>> @@ -36,14 +36,17 @@ static ::testing::AssertionResult NoErro
>>>> namespace {
>>>>
>>>> struct SampleProfTest : ::testing::Test {
>>>> - std::string Data;
>>>> LLVMContext Context;
>>>> + std::string Profile;
>>>> std::unique_ptr<raw_ostream> OS;
>>>> std::unique_ptr<SampleProfileWriter> Writer;
>>>> std::unique_ptr<SampleProfileReader> Reader;
>>>> + std::error_code EC;
>>>>
>>>> SampleProfTest()
>>>> - : Data(), OS(new raw_string_ostream(Data)), Writer(), Reader() {}
>>>> + : Profile("profile"),
>>>> + OS(new raw_fd_ostream(Profile, EC, sys::fs::F_None)), Writer(),
>>>> + Reader() {}
>>>>
>>>> void createWriter(SampleProfileFormat Format) {
>>>> auto WriterOrErr = SampleProfileWriter::create(OS, Format);
>>>> @@ -51,10 +54,11 @@ struct SampleProfTest : ::testing::Test
>>>> Writer = std::move(WriterOrErr.get());
>>>> }
>>>>
>>>> - void readProfile(std::unique_ptr<MemoryBuffer> &Profile) {
>>>> + void readProfile(const Module &M) {
>>>> auto ReaderOrErr = SampleProfileReader::create(Profile, Context);
>>>> ASSERT_TRUE(NoError(ReaderOrErr.getError()));
>>>> Reader = std::move(ReaderOrErr.get());
>>>> + Reader->collectFuncsToUse(M);
>>>> }
>>>>
>>>> void testRoundTrip(SampleProfileFormat Format) {
>>>> @@ -83,6 +87,12 @@ struct SampleProfTest : ::testing::Test
>>>> BarSamples.addCalledTargetSamples(1, 0, MconstructName, 1000);
>>>> BarSamples.addCalledTargetSamples(1, 0, StringviewName, 437);
>>>>
>>>> + Module M("my_module", Context);
>>>> + FunctionType *fn_type =
>>>> + FunctionType::get(Type::getVoidTy(Context), {}, false);
>>>> + M.getOrInsertFunction(FooName, fn_type);
>>>> + M.getOrInsertFunction(BarName, fn_type);
>>>> +
>>>> StringMap<FunctionSamples> Profiles;
>>>> Profiles[FooName] = std::move(FooSamples);
>>>> Profiles[BarName] = std::move(BarSamples);
>>>> @@ -93,8 +103,7 @@ struct SampleProfTest : ::testing::Test
>>>>
>>>> Writer->getOutputStream().flush();
>>>>
>>>> - auto Profile = MemoryBuffer::getMemBufferCopy(Data);
>>>> - readProfile(Profile);
>>>> + readProfile(M);
>>>>
>>>> EC = Reader->read();
>>>> ASSERT_TRUE(NoError(EC));
>>>> @@ -164,7 +173,6 @@ struct SampleProfTest : ::testing::Test
>>>> delete PS;
>>>>
>>>> // Test that summary can be attached to and read back from module.
>>>> - Module M("my_module", Context);
>>>> M.setProfileSummary(MD);
>>>> MD = M.getProfileSummary();
>>>> ASSERT_TRUE(MD);
>>>>
>>>>
>>>> _______________________________________________
>>>> llvm-commits mailing list
>>>> llvm-commits at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>>
>>>
>>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20180917/490c65f7/attachment.html>
More information about the llvm-commits
mailing list