<div dir="ltr">Alexander, thanks for the fix!<div><br></div><div>Wei.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 17, 2018 at 5:12 AM, Alexander Kornienko <span dir="ltr"><<a href="mailto:alexfh@google.com" target="_blank">alexfh@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">Committed a fix in r342386.</div></div><div class="HOEnZb"><div class="h5"><br><div class="gmail_quote"><div dir="ltr">On Mon, Sep 17, 2018 at 2:11 PM Alexander Kornienko <<a href="mailto:alexfh@google.com" target="_blank">alexfh@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">It looks like the code is using seek(), which is not available for raw_string_ostream and raw_svector_ostream. But llvm::sys::fs::<wbr>createTemporaryFile makes this test work for us.</div><br><div class="gmail_quote"><div dir="ltr">On Mon, Sep 17, 2018 at 12:38 PM Alexander Kornienko <<a href="mailto:alexfh@google.com" target="_blank">alexfh@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Another problem with this test is that it tries to write a file in the current directory, which may be read-only in some setups. Is there a reason to access the real file system in this test? Would you mind me switching this to in-memory streams?</div><div dir="ltr"></div><br><div class="gmail_quote"><div dir="ltr">On Sat, Sep 15, 2018 at 2:06 AM Wei Mi via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">Should be fixed in rL342301. </div><div dir="ltr"><br></div><div dir="ltr">Thanks!</div><div dir="ltr">Wei.<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 14, 2018 at 4:35 PM, Wei Mi <span dir="ltr"><<a href="mailto:wmi@google.com" target="_blank">wmi@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">There is file system race indeed. I will fix that. Thanks Reid!<span><font color="#888888"><div><br></div><div>Wei.</div></font></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 14, 2018 at 4:09 PM, Reid Kleckner <span dir="ltr"><<a href="mailto:rnk@google.com" target="_blank">rnk@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">These tests seem flaky when I run them locally. Are there possible filesystem race issues here?<div><br></div><div><div>FAIL: LLVM-Unit :: ProfileData/./<wbr>ProfileDataTests.exe/<wbr>SampleProfTest.roundtrip_<wbr>compact_binary_profile (2329 of 27766)</div><div>******************** TEST 'LLVM-Unit :: ProfileData/./<wbr>ProfileDataTests.exe/<wbr>SampleProfTest.roundtrip_<wbr>compact_binary_profile' FAILED ********************</div><div>Note: Google Test filter = SampleProfTest.roundtrip_<wbr>compact_binary_profile</div><div>[==========] Running 1 test from 1 test case.</div><div>[----------] Global test environment set-up.</div><div>[----------] 1 test from SampleProfTest</div><div>[ RUN ] SampleProfTest.roundtrip_<wbr>compact_binary_profile</div><div>C:\src\llvm-project\llvm\<wbr>unittests\ProfileData\<wbr>SampleProfTest.cpp(59): error: Value of: NoError(ReaderOrErr.getError()<wbr>)</div><div> Actual: false (error 6: Unrecognized sample profile encoding format)</div><div>Expected: true</div><div>unknown file: error: SEH exception with code 0x3221225477 thrown in the test body.</div><div>unknown file: error: SEH exception with code 0x3221225477 thrown in the test fixture's destructor.</div><div>[ FAILED ] SampleProfTest.roundtrip_<wbr>compact_binary_profile (2 ms)</div><div>[----------] 1 test from SampleProfTest (2 ms total)</div><div><br></div><div>[----------] Global test environment tear-down</div><div>[==========] 1 test from 1 test case ran. (2 ms total)</div><div>[ PASSED ] 0 tests.</div><div>[ FAILED ] 1 test, listed below:</div><div>[ FAILED ] SampleProfTest.roundtrip_<wbr>compact_binary_profile</div><div><br></div><div> 1 FAILED TEST</div><div><br></div><div>********************</div><div>Testing: 0 ..</div><div>FAIL: LLVM-Unit :: ProfileData/./<wbr>ProfileDataTests.exe/<wbr>SampleProfTest.roundtrip_text_<wbr>profile (2330 of 27766)</div><div>******************** TEST 'LLVM-Unit :: ProfileData/./<wbr>ProfileDataTests.exe/<wbr>SampleProfTest.roundtrip_text_<wbr>profile' FAILED ********************</div><div>Note: Google Test filter = SampleProfTest.roundtrip_text_<wbr>profile</div><div>[==========] Running 1 test from 1 test case.</div><div>[----------] Global test environment set-up.</div><div>[----------] 1 test from SampleProfTest</div><div>[ RUN ] SampleProfTest.roundtrip_text_<wbr>profile</div><div>C:\src\llvm-project\llvm\<wbr>unittests\ProfileData\<wbr>SampleProfTest.cpp(59): error: Value of: NoError(ReaderOrErr.getError()<wbr>)</div><div> Actual: false (error 6: Unrecognized sample profile encoding format)</div><div>Expected: true</div><div>unknown file: error: SEH exception with code 0x3221225477 thrown in the test body.</div><div>unknown file: error: SEH exception with code 0x3221225477 thrown in the test fixture's destructor.</div><div>[ FAILED ] SampleProfTest.roundtrip_text_<wbr>profile (2 ms)</div><div>[----------] 1 test from SampleProfTest (2 ms total)</div><div><br></div><div>[----------] Global test environment tear-down</div><div>[==========] 1 test from 1 test case ran. (2 ms total)</div><div>[ PASSED ] 0 tests.</div><div>[ FAILED ] 1 test, listed below:</div><div>[ FAILED ] SampleProfTest.roundtrip_text_<wbr>profile</div><div><br></div><div> 1 FAILED TEST</div><div><br></div><div>********************</div></div></div></div><div><div><br><div class="gmail_quote"><div dir="ltr">On Fri, Sep 14, 2018 at 1:54 PM Wei Mi via llvm-commits <<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Author: wmi<br>
Date: Fri Sep 14 13:52:59 2018<br>
New Revision: 342283<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=342283&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project?rev=342283&view=rev</a><br>
Log:<br>
[SampleFDO] Add FunctionOffsetTable in compact binary format profile.<br>
<br>
The patch saves a function offset table which maps function name index to the<br>
offset of its function profile to the start of the binary profile. By using<br>
the function offset table, for those function profiles which will not be used<br>
when compiling a module, the profile reader does't have to read them. For<br>
profile size around 10~20M, it saves ~10% compile time.<br>
<br>
Differential Revision: <a href="https://reviews.llvm.org/D51863" rel="noreferrer" target="_blank">https://reviews.llvm.org/<wbr>D51863</a><br>
<br>
<br>
Modified:<br>
llvm/trunk/include/llvm/<wbr>ProfileData/SampleProf.h<br>
llvm/trunk/include/llvm/<wbr>ProfileData/SampleProfReader.h<br>
llvm/trunk/include/llvm/<wbr>ProfileData/SampleProfWriter.h<br>
llvm/trunk/lib/ProfileData/<wbr>SampleProf.cpp<br>
llvm/trunk/lib/ProfileData/<wbr>SampleProfReader.cpp<br>
llvm/trunk/lib/ProfileData/<wbr>SampleProfWriter.cpp<br>
llvm/trunk/lib/Transforms/IPO/<wbr>SampleProfile.cpp<br>
llvm/trunk/test/Transforms/<wbr>SampleProfile/Inputs/function_<wbr>metadata.compact.afdo<br>
llvm/trunk/test/Transforms/<wbr>SampleProfile/Inputs/indirect-<wbr>call.compact.afdo<br>
llvm/trunk/test/Transforms/<wbr>SampleProfile/Inputs/inline.<wbr>compactbinary.afdo<br>
llvm/trunk/unittests/<wbr>ProfileData/SampleProfTest.cpp<br>
<br>
Modified: llvm/trunk/include/llvm/<wbr>ProfileData/SampleProf.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/ProfileData/SampleProf.h?rev=342283&r1=342282&r2=342283&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/include/<wbr>llvm/ProfileData/SampleProf.h?<wbr>rev=342283&r1=342282&r2=<wbr>342283&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/include/llvm/<wbr>ProfileData/SampleProf.h (original)<br>
+++ llvm/trunk/include/llvm/<wbr>ProfileData/SampleProf.h Fri Sep 14 13:52:59 2018<br>
@@ -49,7 +49,8 @@ enum class sampleprof_error {<br>
unsupported_writing_format,<br>
truncated_name_table,<br>
not_implemented,<br>
- counter_overflow<br>
+ counter_overflow,<br>
+ ostream_seek_unsupported<br>
};<br>
<br>
inline std::error_code make_error_code(sampleprof_<wbr>error E) {<br>
<br>
Modified: llvm/trunk/include/llvm/<wbr>ProfileData/SampleProfReader.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/ProfileData/SampleProfReader.h?rev=342283&r1=342282&r2=342283&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/include/<wbr>llvm/ProfileData/<wbr>SampleProfReader.h?rev=342283&<wbr>r1=342282&r2=342283&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/include/llvm/<wbr>ProfileData/SampleProfReader.h (original)<br>
+++ llvm/trunk/include/llvm/<wbr>ProfileData/SampleProfReader.h Fri Sep 14 13:52:59 2018<br>
@@ -279,6 +279,8 @@ public:<br>
/// Print the profile for \p FName on stream \p OS.<br>
void dumpFunctionProfile(StringRef FName, raw_ostream &OS = dbgs());<br>
<br>
+ virtual void collectFuncsToUse(const Module &M) {}<br>
+<br>
/// Print all the profiles on stream \p OS.<br>
void dump(raw_ostream &OS = dbgs());<br>
<br>
@@ -364,7 +366,7 @@ public:<br>
: SampleProfileReader(std::move(<wbr>B), C, Format) {}<br>
<br>
/// Read and validate the file header.<br>
- std::error_code readHeader() override;<br>
+ virtual std::error_code readHeader() override;<br>
<br>
/// Read sample profiles from the associated file.<br>
std::error_code read() override;<br>
@@ -378,6 +380,10 @@ protected:<br>
/// \returns the read value.<br>
template <typename T> ErrorOr<T> readNumber();<br>
<br>
+ /// Read a numeric value of type T from the profile. The value is saved<br>
+ /// without encoded.<br>
+ template <typename T> ErrorOr<T> readUnencodedNumber();<br>
+<br>
/// Read a string from the profile.<br>
///<br>
/// If an error occurs during decoding, a diagnostic message is emitted and<br>
@@ -392,6 +398,9 @@ protected:<br>
/// Return true if we've reached the end of file.<br>
bool at_eof() const { return Data >= End; }<br>
<br>
+ /// Read the next function profile instance.<br>
+ std::error_code readFuncProfile();<br>
+<br>
/// Read the contents of the given profile instance.<br>
std::error_code readProfile(FunctionSamples &FProfile);<br>
<br>
@@ -436,10 +445,17 @@ class SampleProfileReaderCompactBina<wbr>ry :<br>
private:<br>
/// Function name table.<br>
std::vector<std::string> NameTable;<br>
+ /// The table mapping from function name to the offset of its FunctionSample<br>
+ /// towards file start.<br>
+ DenseMap<StringRef, uint64_t> FuncOffsetTable;<br>
+ /// The set containing the functions to use when compiling a module.<br>
+ DenseSet<StringRef> FuncsToUse;<br>
virtual std::error_code verifySPMagic(uint64_t Magic) override;<br>
virtual std::error_code readNameTable() override;<br>
/// Read a string indirectly via the name table.<br>
virtual ErrorOr<StringRef> readStringFromTable() override;<br>
+ virtual std::error_code readHeader() override;<br>
+ std::error_code readFuncOffsetTable();<br>
<br>
public:<br>
<wbr>SampleProfileReaderCompactBina<wbr>ry(std::unique_ptr<<wbr>MemoryBuffer> B,<br>
@@ -448,6 +464,12 @@ public:<br>
<br>
/// \brief Return true if \p Buffer is in the format supported by this class.<br>
static bool hasFormat(const MemoryBuffer &Buffer);<br>
+<br>
+ /// Read samples only for functions to use.<br>
+ std::error_code read() override;<br>
+<br>
+ /// Collect functions to be used when compiling Module \p M.<br>
+ void collectFuncsToUse(const Module &M) override;<br>
};<br>
<br>
using InlineCallStack = SmallVector<FunctionSamples *, 10>;<br>
<br>
Modified: llvm/trunk/include/llvm/<wbr>ProfileData/SampleProfWriter.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/ProfileData/SampleProfWriter.h?rev=342283&r1=342282&r2=342283&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/include/<wbr>llvm/ProfileData/<wbr>SampleProfWriter.h?rev=342283&<wbr>r1=342282&r2=342283&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/include/llvm/<wbr>ProfileData/SampleProfWriter.h (original)<br>
+++ llvm/trunk/include/llvm/<wbr>ProfileData/SampleProfWriter.h Fri Sep 14 13:52:59 2018<br>
@@ -42,7 +42,7 @@ public:<br>
/// Write all the sample profiles in the given map of samples.<br>
///<br>
/// \returns status code of the file update operation.<br>
- std::error_code write(const StringMap<FunctionSamples> &ProfileMap);<br>
+ virtual std::error_code write(const StringMap<FunctionSamples> &ProfileMap);<br>
<br>
raw_ostream &getOutputStream() { return *OutputStream; }<br>
<br>
@@ -103,14 +103,15 @@ private:<br>
/// Sample-based profile writer (binary format).<br>
class SampleProfileWriterBinary : public SampleProfileWriter {<br>
public:<br>
- std::error_code write(const FunctionSamples &S) override;<br>
+ virtual std::error_code write(const FunctionSamples &S) override;<br>
SampleProfileWriterBinary(<wbr>std::unique_ptr<raw_ostream> &OS)<br>
: SampleProfileWriter(OS) {}<br>
<br>
protected:<br>
virtual std::error_code writeNameTable() = 0;<br>
virtual std::error_code writeMagicIdent() = 0;<br>
- std::error_code writeHeader(const StringMap<FunctionSamples> &ProfileMap) override;<br>
+ virtual std::error_code<br>
+ writeHeader(const StringMap<FunctionSamples> &ProfileMap) override;<br>
std::error_code writeSummary();<br>
std::error_code writeNameIdx(StringRef FName);<br>
std::error_code writeBody(const FunctionSamples &S);<br>
@@ -135,12 +136,56 @@ protected:<br>
virtual std::error_code writeMagicIdent() override;<br>
};<br>
<br>
+// CompactBinary is a compact format of binary profile which both reduces<br>
+// the profile size and the load time needed when compiling. It has two<br>
+// major difference with Binary format.<br>
+// 1. It represents all the strings in name table using md5 hash.<br>
+// 2. It saves a function offset table which maps function name index to<br>
+// the offset of its function profile to the start of the binary profile,<br>
+// so by using the function offset table, for those function profiles which<br>
+// will not be needed when compiling a module, the profile reader does't<br>
+// have to read them and it saves compile time if the profile size is huge.<br>
+// The layout of the compact format is shown as follows:<br>
+//<br>
+// Part1: Profile header, the same as binary format, containing magic<br>
+// number, version, summary, name table...<br>
+// Part2: Function Offset Table Offset, which saves the position of<br>
+// Part4.<br>
+// Part3: Function profile collection<br>
+// function1 profile start<br>
+// ....<br>
+// function2 profile start<br>
+// ....<br>
+// function3 profile start<br>
+// ....<br>
+// ......<br>
+// Part4: Function Offset Table<br>
+// function1 name index --> function1 profile start<br>
+// function2 name index --> function2 profile start<br>
+// function3 name index --> function3 profile start<br>
+//<br>
+// We need Part2 because profile reader can use it to find out and read<br>
+// function offset table without reading Part3 first.<br>
class SampleProfileWriterCompactBina<wbr>ry : public SampleProfileWriterBinary {<br>
using SampleProfileWriterBinary::<wbr>SampleProfileWriterBinary;<br>
<br>
+public:<br>
+ virtual std::error_code write(const FunctionSamples &S) override;<br>
+ virtual std::error_code<br>
+ write(const StringMap<FunctionSamples> &ProfileMap) override;<br>
+<br>
protected:<br>
+ /// The table mapping from function name to the offset of its FunctionSample<br>
+ /// towards profile start.<br>
+ MapVector<StringRef, uint64_t> FuncOffsetTable;<br>
+ /// The offset of the slot to be filled with the offset of FuncOffsetTable<br>
+ /// towards profile start.<br>
+ uint64_t TableOffset;<br>
virtual std::error_code writeNameTable() override;<br>
virtual std::error_code writeMagicIdent() override;<br>
+ virtual std::error_code<br>
+ writeHeader(const StringMap<FunctionSamples> &ProfileMap) override;<br>
+ std::error_code writeFuncOffsetTable();<br>
};<br>
<br>
} // end namespace sampleprof<br>
<br>
Modified: llvm/trunk/lib/ProfileData/<wbr>SampleProf.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/ProfileData/SampleProf.cpp?rev=342283&r1=342282&r2=342283&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/lib/<wbr>ProfileData/SampleProf.cpp?<wbr>rev=342283&r1=342282&r2=<wbr>342283&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/lib/ProfileData/<wbr>SampleProf.cpp (original)<br>
+++ llvm/trunk/lib/ProfileData/<wbr>SampleProf.cpp Fri Sep 14 13:52:59 2018<br>
@@ -67,6 +67,8 @@ class SampleProfErrorCategoryType : publ<br>
return "Unimplemented feature";<br>
case sampleprof_error::counter_<wbr>overflow:<br>
return "Counter overflow";<br>
+ case sampleprof_error::ostream_<wbr>seek_unsupported:<br>
+ return "Ostream does not support seek";<br>
}<br>
llvm_unreachable("A value of sampleprof_error has no message.");<br>
}<br>
<br>
Modified: llvm/trunk/lib/ProfileData/<wbr>SampleProfReader.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/ProfileData/SampleProfReader.cpp?rev=342283&r1=342282&r2=342283&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/lib/<wbr>ProfileData/SampleProfReader.<wbr>cpp?rev=342283&r1=342282&r2=<wbr>342283&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/lib/ProfileData/<wbr>SampleProfReader.cpp (original)<br>
+++ llvm/trunk/lib/ProfileData/<wbr>SampleProfReader.cpp Fri Sep 14 13:52:59 2018<br>
@@ -30,6 +30,7 @@<br>
#include "llvm/Support/ErrorOr.h"<br>
#include "llvm/Support/LEB128.h"<br>
#include "llvm/Support/LineIterator.h"<br>
+#include "llvm/Support/MD5.h"<br>
#include "llvm/Support/MemoryBuffer.h"<br>
#include "llvm/Support/raw_ostream.h"<br>
#include <algorithm><br>
@@ -320,6 +321,21 @@ ErrorOr<StringRef> SampleProfileReaderBi<br>
}<br>
<br>
template <typename T><br>
+ErrorOr<T> SampleProfileReaderBinary::<wbr>readUnencodedNumber() {<br>
+ std::error_code EC;<br>
+<br>
+ if (Data + sizeof(T) > End) {<br>
+ EC = sampleprof_error::truncated;<br>
+ reportError(0, EC.message());<br>
+ return EC;<br>
+ }<br>
+<br>
+ using namespace support;<br>
+ T Val = endian::readNext<T, little, unaligned>(Data);<br>
+ return Val;<br>
+}<br>
+<br>
+template <typename T><br>
inline ErrorOr<uint32_t> SampleProfileReaderBinary::<wbr>readStringIndex(T &Table) {<br>
std::error_code EC;<br>
auto Idx = readNumber<uint32_t>();<br>
@@ -423,29 +439,51 @@ SampleProfileReaderBinary::<wbr>readProfile(F<br>
return sampleprof_error::success;<br>
}<br>
<br>
-std::error_code SampleProfileReaderBinary::<wbr>read() {<br>
- while (!at_eof()) {<br>
- auto NumHeadSamples = readNumber<uint64_t>();<br>
- if (std::error_code EC = NumHeadSamples.getError())<br>
- return EC;<br>
+std::error_code SampleProfileReaderBinary::<wbr>readFuncProfile() {<br>
+ auto NumHeadSamples = readNumber<uint64_t>();<br>
+ if (std::error_code EC = NumHeadSamples.getError())<br>
+ return EC;<br>
<br>
- auto FName(readStringFromTable());<br>
- if (std::error_code EC = FName.getError())<br>
- return EC;<br>
+ auto FName(readStringFromTable());<br>
+ if (std::error_code EC = FName.getError())<br>
+ return EC;<br>
<br>
- Profiles[*FName] = FunctionSamples();<br>
- FunctionSamples &FProfile = Profiles[*FName];<br>
- FProfile.setName(*FName);<br>
+ Profiles[*FName] = FunctionSamples();<br>
+ FunctionSamples &FProfile = Profiles[*FName];<br>
+ FProfile.setName(*FName);<br>
<br>
- FProfile.addHeadSamples(*<wbr>NumHeadSamples);<br>
+ FProfile.addHeadSamples(*<wbr>NumHeadSamples);<br>
+<br>
+ if (std::error_code EC = readProfile(FProfile))<br>
+ return EC;<br>
+ return sampleprof_error::success;<br>
+}<br>
<br>
- if (std::error_code EC = readProfile(FProfile))<br>
+std::error_code SampleProfileReaderBinary::<wbr>read() {<br>
+ while (!at_eof()) {<br>
+ if (std::error_code EC = readFuncProfile())<br>
return EC;<br>
}<br>
<br>
return sampleprof_error::success;<br>
}<br>
<br>
+std::error_code SampleProfileReaderCompactBina<wbr>ry::read() {<br>
+ for (auto Name : FuncsToUse) {<br>
+ auto GUID = std::to_string(MD5Hash(Name));<br>
+ auto iter = FuncOffsetTable.find(<wbr>StringRef(GUID));<br>
+ if (iter == FuncOffsetTable.end())<br>
+ continue;<br>
+ const uint8_t *SavedData = Data;<br>
+ Data = reinterpret_cast<const uint8_t *>(Buffer->getBufferStart()) +<br>
+ iter->second;<br>
+ if (std::error_code EC = readFuncProfile())<br>
+ return EC;<br>
+ Data = SavedData;<br>
+ }<br>
+ return sampleprof_error::success;<br>
+}<br>
+<br>
std::error_code SampleProfileReaderRawBinary::<wbr>verifySPMagic(uint64_t Magic) {<br>
if (Magic == SPMagic())<br>
return sampleprof_error::success;<br>
@@ -514,6 +552,53 @@ std::error_code SampleProfileReaderBinar<br>
return sampleprof_error::success;<br>
}<br>
<br>
+std::error_code SampleProfileReaderCompactBina<wbr>ry::readHeader() {<br>
+ SampleProfileReaderBinary::<wbr>readHeader();<br>
+ if (std::error_code EC = readFuncOffsetTable())<br>
+ return EC;<br>
+ return sampleprof_error::success;<br>
+}<br>
+<br>
+std::error_code SampleProfileReaderCompactBina<wbr>ry::readFuncOffsetTable() {<br>
+ auto TableOffset = readUnencodedNumber<uint64_t>(<wbr>);<br>
+ if (std::error_code EC = TableOffset.getError())<br>
+ return EC;<br>
+<br>
+ const uint8_t *SavedData = Data;<br>
+ const uint8_t *TableStart =<br>
+ reinterpret_cast<const uint8_t *>(Buffer->getBufferStart()) +<br>
+ *TableOffset;<br>
+ Data = TableStart;<br>
+<br>
+ auto Size = readNumber<uint64_t>();<br>
+ if (std::error_code EC = Size.getError())<br>
+ return EC;<br>
+<br>
+ FuncOffsetTable.reserve(*Size)<wbr>;<br>
+ for (uint32_t I = 0; I < *Size; ++I) {<br>
+ auto FName(readStringFromTable());<br>
+ if (std::error_code EC = FName.getError())<br>
+ return EC;<br>
+<br>
+ auto Offset = readNumber<uint64_t>();<br>
+ if (std::error_code EC = Offset.getError())<br>
+ return EC;<br>
+<br>
+ FuncOffsetTable[*FName] = *Offset;<br>
+ }<br>
+ End = TableStart;<br>
+ Data = SavedData;<br>
+ return sampleprof_error::success;<br>
+}<br>
+<br>
+void SampleProfileReaderCompactBina<wbr>ry::collectFuncsToUse(const Module &M) {<br>
+ FuncsToUse.clear();<br>
+ for (auto &F : M) {<br>
+ StringRef Fname = F.getName().split('.').first;<br>
+ FuncsToUse.insert(Fname);<br>
+ }<br>
+}<br>
+<br>
std::error_code SampleProfileReaderBinary::<wbr>readSummaryEntry(<br>
std::vector<<wbr>ProfileSummaryEntry> &Entries) {<br>
auto Cutoff = readNumber<uint64_t>();<br>
<br>
Modified: llvm/trunk/lib/ProfileData/<wbr>SampleProfWriter.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/ProfileData/SampleProfWriter.cpp?rev=342283&r1=342282&r2=342283&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/lib/<wbr>ProfileData/SampleProfWriter.<wbr>cpp?rev=342283&r1=342282&r2=<wbr>342283&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/lib/ProfileData/<wbr>SampleProfWriter.cpp (original)<br>
+++ llvm/trunk/lib/ProfileData/<wbr>SampleProfWriter.cpp Fri Sep 14 13:52:59 2018<br>
@@ -22,6 +22,8 @@<br>
#include "llvm/ADT/StringRef.h"<br>
#include "llvm/ProfileData/<wbr>ProfileCommon.h"<br>
#include "llvm/ProfileData/SampleProf.<wbr>h"<br>
+#include "llvm/Support/Endian.h"<br>
+#include "llvm/Support/EndianStream.h"<br>
#include "llvm/Support/ErrorOr.h"<br>
#include "llvm/Support/FileSystem.h"<br>
#include "llvm/Support/LEB128.h"<br>
@@ -64,6 +66,15 @@ SampleProfileWriter::write(<wbr>const StringM<br>
return sampleprof_error::success;<br>
}<br>
<br>
+std::error_code SampleProfileWriterCompactBina<wbr>ry::write(<br>
+ const StringMap<FunctionSamples> &ProfileMap) {<br>
+ if (std::error_code EC = SampleProfileWriter::write(<wbr>ProfileMap))<br>
+ return EC;<br>
+ if (std::error_code EC = writeFuncOffsetTable())<br>
+ return EC;<br>
+ return sampleprof_error::success;<br>
+}<br>
+<br>
/// Write samples to a text file.<br>
///<br>
/// Note: it may be tempting to implement this in terms of<br>
@@ -168,6 +179,30 @@ std::error_code SampleProfileWriterRawBi<br>
return sampleprof_error::success;<br>
}<br>
<br>
+std::error_code SampleProfileWriterCompactBina<wbr>ry::writeFuncOffsetTable() {<br>
+ auto &OS = *OutputStream;<br>
+<br>
+ // Fill the slot remembered by TableOffset with the offset of FuncOffsetTable.<br>
+ auto &OFS = static_cast<raw_fd_ostream &>(OS);<br>
+ uint64_t FuncOffsetTableStart = OS.tell();<br>
+ if (OFS.seek(TableOffset) == (uint64_t)-1)<br>
+ return sampleprof_error::ostream_<wbr>seek_unsupported;<br>
+ support::endian::Writer Writer(*OutputStream, support::little);<br>
+ Writer.write(<wbr>FuncOffsetTableStart);<br>
+ if (OFS.seek(<wbr>FuncOffsetTableStart) == (uint64_t)-1)<br>
+ return sampleprof_error::ostream_<wbr>seek_unsupported;<br>
+<br>
+ // Write out the table size.<br>
+ encodeULEB128(FuncOffsetTable.<wbr>size(), OS);<br>
+<br>
+ // Write out FuncOffsetTable.<br>
+ for (auto entry : FuncOffsetTable) {<br>
+ writeNameIdx(entry.first);<br>
+ encodeULEB128(entry.second, OS);<br>
+ }<br>
+ return sampleprof_error::success;<br>
+}<br>
+<br>
std::error_code SampleProfileWriterCompactBina<wbr>ry::writeNameTable() {<br>
auto &OS = *OutputStream;<br>
std::set<StringRef> V;<br>
@@ -215,6 +250,19 @@ std::error_code SampleProfileWriterBinar<br>
return sampleprof_error::success;<br>
}<br>
<br>
+std::error_code SampleProfileWriterCompactBina<wbr>ry::writeHeader(<br>
+ const StringMap<FunctionSamples> &ProfileMap) {<br>
+ support::endian::Writer Writer(*OutputStream, support::little);<br>
+ if (auto EC = SampleProfileWriterBinary::<wbr>writeHeader(ProfileMap))<br>
+ return EC;<br>
+<br>
+ // Reserve a slot for the offset of function offset table. The slot will<br>
+ // be populated with the offset of FuncOffsetTable later.<br>
+ TableOffset = OutputStream->tell();<br>
+ Writer.write(static_cast<<wbr>uint64_t>(-2));<br>
+ return sampleprof_error::success;<br>
+}<br>
+<br>
std::error_code SampleProfileWriterBinary::<wbr>writeSummary() {<br>
auto &OS = *OutputStream;<br>
encodeULEB128(Summary-><wbr>getTotalCount(), OS);<br>
@@ -282,6 +330,15 @@ std::error_code SampleProfileWriterBinar<br>
encodeULEB128(S.<wbr>getHeadSamples(), *OutputStream);<br>
return writeBody(S);<br>
}<br>
+<br>
+std::error_code<br>
+<wbr>SampleProfileWriterCompactBina<wbr>ry::write(const FunctionSamples &S) {<br>
+ uint64_t Offset = OutputStream->tell();<br>
+ StringRef Name = S.getName();<br>
+ FuncOffsetTable[Name] = Offset;<br>
+ encodeULEB128(S.<wbr>getHeadSamples(), *OutputStream);<br>
+ return writeBody(S);<br>
+}<br>
<br>
/// Create a sample profile file writer based on the specified format.<br>
///<br>
<br>
Modified: llvm/trunk/lib/Transforms/IPO/<wbr>SampleProfile.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Transforms/IPO/SampleProfile.cpp?rev=342283&r1=342282&r2=342283&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/lib/<wbr>Transforms/IPO/SampleProfile.<wbr>cpp?rev=342283&r1=342282&r2=<wbr>342283&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/lib/Transforms/IPO/<wbr>SampleProfile.cpp (original)<br>
+++ llvm/trunk/lib/Transforms/IPO/<wbr>SampleProfile.cpp Fri Sep 14 13:52:59 2018<br>
@@ -1515,6 +1515,7 @@ bool SampleProfileLoader::<wbr>doInitializati<br>
return false;<br>
}<br>
Reader = std::move(ReaderOrErr.get());<br>
+ Reader->collectFuncsToUse(M);<br>
ProfileIsValid = (Reader->read() == sampleprof_error::success);<br>
return true;<br>
}<br>
<br>
Modified: llvm/trunk/test/Transforms/<wbr>SampleProfile/Inputs/function_<wbr>metadata.compact.afdo<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SampleProfile/Inputs/function_metadata.compact.afdo?rev=342283&r1=342282&r2=342283&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/test/<wbr>Transforms/SampleProfile/<wbr>Inputs/function_metadata.<wbr>compact.afdo?rev=342283&r1=<wbr>342282&r2=342283&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
Binary files - no diff available.<br>
<br>
Modified: llvm/trunk/test/Transforms/<wbr>SampleProfile/Inputs/indirect-<wbr>call.compact.afdo<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SampleProfile/Inputs/indirect-call.compact.afdo?rev=342283&r1=342282&r2=342283&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/test/<wbr>Transforms/SampleProfile/<wbr>Inputs/indirect-call.compact.<wbr>afdo?rev=342283&r1=342282&r2=<wbr>342283&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
Binary files - no diff available.<br>
<br>
Modified: llvm/trunk/test/Transforms/<wbr>SampleProfile/Inputs/inline.<wbr>compactbinary.afdo<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Transforms/SampleProfile/Inputs/inline.compactbinary.afdo?rev=342283&r1=342282&r2=342283&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/test/<wbr>Transforms/SampleProfile/<wbr>Inputs/inline.compactbinary.<wbr>afdo?rev=342283&r1=342282&r2=<wbr>342283&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
Binary files - no diff available.<br>
<br>
Modified: llvm/trunk/unittests/<wbr>ProfileData/SampleProfTest.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/unittests/ProfileData/SampleProfTest.cpp?rev=342283&r1=342282&r2=342283&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/unittests/<wbr>ProfileData/SampleProfTest.<wbr>cpp?rev=342283&r1=342282&r2=<wbr>342283&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/unittests/<wbr>ProfileData/SampleProfTest.cpp (original)<br>
+++ llvm/trunk/unittests/<wbr>ProfileData/SampleProfTest.cpp Fri Sep 14 13:52:59 2018<br>
@@ -36,14 +36,17 @@ static ::testing::AssertionResult NoErro<br>
namespace {<br>
<br>
struct SampleProfTest : ::testing::Test {<br>
- std::string Data;<br>
LLVMContext Context;<br>
+ std::string Profile;<br>
std::unique_ptr<raw_ostream> OS;<br>
std::unique_ptr<<wbr>SampleProfileWriter> Writer;<br>
std::unique_ptr<<wbr>SampleProfileReader> Reader;<br>
+ std::error_code EC;<br>
<br>
SampleProfTest()<br>
- : Data(), OS(new raw_string_ostream(Data)), Writer(), Reader() {}<br>
+ : Profile("profile"),<br>
+ OS(new raw_fd_ostream(Profile, EC, sys::fs::F_None)), Writer(),<br>
+ Reader() {}<br>
<br>
void createWriter(<wbr>SampleProfileFormat Format) {<br>
auto WriterOrErr = SampleProfileWriter::create(<wbr>OS, Format);<br>
@@ -51,10 +54,11 @@ struct SampleProfTest : ::testing::Test<br>
Writer = std::move(WriterOrErr.get());<br>
}<br>
<br>
- void readProfile(std::unique_ptr<<wbr>MemoryBuffer> &Profile) {<br>
+ void readProfile(const Module &M) {<br>
auto ReaderOrErr = SampleProfileReader::create(<wbr>Profile, Context);<br>
ASSERT_TRUE(NoError(<wbr>ReaderOrErr.getError()));<br>
Reader = std::move(ReaderOrErr.get());<br>
+ Reader->collectFuncsToUse(M);<br>
}<br>
<br>
void testRoundTrip(<wbr>SampleProfileFormat Format) {<br>
@@ -83,6 +87,12 @@ struct SampleProfTest : ::testing::Test<br>
BarSamples.<wbr>addCalledTargetSamples(1, 0, MconstructName, 1000);<br>
BarSamples.<wbr>addCalledTargetSamples(1, 0, StringviewName, 437);<br>
<br>
+ Module M("my_module", Context);<br>
+ FunctionType *fn_type =<br>
+ FunctionType::get(Type::<wbr>getVoidTy(Context), {}, false);<br>
+ M.getOrInsertFunction(FooName, fn_type);<br>
+ M.getOrInsertFunction(BarName, fn_type);<br>
+<br>
StringMap<FunctionSamples> Profiles;<br>
Profiles[FooName] = std::move(FooSamples);<br>
Profiles[BarName] = std::move(BarSamples);<br>
@@ -93,8 +103,7 @@ struct SampleProfTest : ::testing::Test<br>
<br>
Writer->getOutputStream().<wbr>flush();<br>
<br>
- auto Profile = MemoryBuffer::<wbr>getMemBufferCopy(Data);<br>
- readProfile(Profile);<br>
+ readProfile(M);<br>
<br>
EC = Reader->read();<br>
ASSERT_TRUE(NoError(EC));<br>
@@ -164,7 +173,6 @@ struct SampleProfTest : ::testing::Test<br>
delete PS;<br>
<br>
// Test that summary can be attached to and read back from module.<br>
- Module M("my_module", Context);<br>
M.setProfileSummary(MD);<br>
MD = M.getProfileSummary();<br>
ASSERT_TRUE(MD);<br>
<br>
<br>
______________________________<wbr>_________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/llvm-commits</a><br>
</blockquote></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
______________________________<wbr>_________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/llvm-commits</a><br>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</div></div></blockquote></div><br></div>