[llvm] 52c583b - [SampleFDO][TypeProf]Support vtable type profiling for ext-binary and text format (#148002)
via llvm-commits
llvm-commits at lists.llvm.org
Fri Sep 12 15:58:20 PDT 2025
Author: Mingming Liu
Date: 2025-09-12T15:58:16-07:00
New Revision: 52c583b3f95a0e666ab837e39a5db900b66adf15
URL: https://github.com/llvm/llvm-project/commit/52c583b3f95a0e666ab837e39a5db900b66adf15
DIFF: https://github.com/llvm/llvm-project/commit/52c583b3f95a0e666ab837e39a5db900b66adf15.diff
LOG: [SampleFDO][TypeProf]Support vtable type profiling for ext-binary and text format (#148002)
This change extends SampleFDO ext-binary and text format to record the
vtable symbols and their counts for virtual calls inside a function. The
vtable profiles will allow the compiler to annotate vtable types on IR
instructions and perform vtable-based indirect call promotion. An RFC is
in
https://discourse.llvm.org/t/rfc-vtable-type-profiling-for-samplefdo/87283
Given a function below, the before vs after of a function's profile is
illustrated in text format in the table:
```
__attribute__((noinline)) int loop_func(int i, int a, int b) {
Base *ptr = createType(i);
int sum = ptr->func(a, b);
delete ptr;
return sum;
}
```
| before | after |
| --- | --- |
| Samples collected in the function's body { <br> 0: 636241 <br> 1:
681458, calls: _Z10createTypei:681458 <br> 3: 543499, calls:
_ZN12_GLOBAL__N_18Derived24funcEii:410621 _ZN8Derived14funcEii:132878
<br> 5.1: 602201, calls: _ZN12_GLOBAL__N_18Derived2D0Ev:454635
_ZN8Derived1D0Ev:147566 <br> 7: 511057 <br> } | Samples collected in the
function's body { <br> 0: 636241 <br> 1: 681458, calls:
_Z10createTypei:681458 <br> 3: 543499, calls:
_ZN12_GLOBAL__N_18Derived24funcEii:410621 _ZN8Derived14funcEii:132878
<br> 3: vtables: _ZTV8Derived1:1377 _ZTVN12_GLOBAL__N_18Derived2E:4250
<br> 5.1: 602201, calls: _ZN12_GLOBAL__N_18Derived2D0Ev:454635
_ZN8Derived1D0Ev:147566 <br> 5.1: vtables: _ZTV8Derived1:227
_ZTVN12_GLOBAL__N_18Derived2E:765 <br> 7: 511057 <br> } |
Key points for this change:
1. In-memory representation of vtable profiles
* A field of type `map<LineLocation, map<FunctionId, uint64_t>>` is
introduced in a function's in-memory representation
[FunctionSamples](https://github.com/llvm/llvm-project/blob/ccc416312ed72e92a885425d9cb9c01f9afa58eb/llvm/include/llvm/ProfileData/SampleProf.h#L749-L754).
2. The vtable counters for one LineLocation represents the relative
frequency among vtables for this LineLocation. They are not required to
be comparable across LineLocations.
3. For backward compatibility of ext-binary format, we take one bit from
ProfSummaryFlag as illustrated in the enum class `SecProfSummaryFlags`.
The ext-binary profile reader parses the integer type flag and reads
this bit. If it's set, the profile reader will parse vtable profiles.
4. The vtable profiles are optional in ext-binary format, and not
serialized out by default, we introduce an LLVM boolean option (named
`-extbinary-write-vtable-type-prof`). The ext-binary profile writer
reads the boolean option and decide whether to set the section flag bit
and serialize the in-memory class members corresponding to vtables.
5. This change doesn't implement `llvm-profdata overlap --sample` for
the vtable profiles. A subsequent change will do it to keep this one
focused on the profile format change.
We don't plan to add the vtable support to non-extensible format mainly
because of the maintenance cost to keep backward compatibility for prior
versions of profile data.
* Currently, the [non-extensible binary
format](https://github.com/llvm/llvm-project/blob/5c28af409978c19a35021855a29dcaa65e95da00/llvm/lib/ProfileData/SampleProfWriter.cpp#L899-L900)
does not have feature parity with extensible binary format today, for
instance, the former doesn't support [profile symbol
list](https://github.com/llvm/llvm-project/blob/41e22aa31b1905aa3e9d83c0343a96ec0d5187ec/llvm/include/llvm/ProfileData/SampleProf.h#L1518-L1522)
or context-sensitive PGO, both of which give measurable performance
boost. Presumably the non-extensible format is not in wide use.
---------
Co-authored-by: Paschalis Mpeis <paschalis.mpeis at arm.com>
Added:
llvm/test/tools/llvm-profdata/Inputs/profile-symbol-list-ext.expected
llvm/test/tools/llvm-profdata/Inputs/sample-profile-ext.proftext
Modified:
llvm/include/llvm/ProfileData/SampleProf.h
llvm/include/llvm/ProfileData/SampleProfReader.h
llvm/include/llvm/ProfileData/SampleProfWriter.h
llvm/lib/ProfileData/SampleProf.cpp
llvm/lib/ProfileData/SampleProfReader.cpp
llvm/lib/ProfileData/SampleProfWriter.cpp
llvm/test/tools/llvm-profdata/profile-symbol-list-compress.test
llvm/test/tools/llvm-profdata/profile-symbol-list.test
llvm/test/tools/llvm-profdata/roundtrip.test
Removed:
################################################################################
diff --git a/llvm/include/llvm/ProfileData/SampleProf.h b/llvm/include/llvm/ProfileData/SampleProf.h
index a626071d23915..c0e5d2d79cea2 100644
--- a/llvm/include/llvm/ProfileData/SampleProf.h
+++ b/llvm/include/llvm/ProfileData/SampleProf.h
@@ -62,7 +62,7 @@ enum class sampleprof_error {
uncompress_failed,
zlib_unavailable,
hash_mismatch,
- illegal_line_offset
+ illegal_line_offset,
};
inline std::error_code make_error_code(sampleprof_error E) {
@@ -91,6 +91,8 @@ struct is_error_code_enum<llvm::sampleprof_error> : std::true_type {};
namespace llvm {
namespace sampleprof {
+constexpr char kVTableProfPrefix[] = "vtables ";
+
enum SampleProfileFormat {
SPF_None = 0,
SPF_Text = 0x1,
@@ -204,6 +206,9 @@ enum class SecProfSummaryFlags : uint32_t {
/// SecFlagIsPreInlined means this profile contains ShouldBeInlined
/// contexts thus this is CS preinliner computed.
SecFlagIsPreInlined = (1 << 4),
+
+ /// SecFlagHasVTableTypeProf means this profile contains vtable type profiles.
+ SecFlagHasVTableTypeProf = (1 << 5),
};
enum class SecFuncMetadataFlags : uint32_t {
@@ -303,7 +308,7 @@ struct LineLocation {
}
uint64_t getHashCode() const {
- return ((uint64_t) Discriminator << 32) | LineOffset;
+ return ((uint64_t)Discriminator << 32) | LineOffset;
}
uint32_t LineOffset;
@@ -318,16 +323,30 @@ struct LineLocationHash {
LLVM_ABI raw_ostream &operator<<(raw_ostream &OS, const LineLocation &Loc);
+/// Key represents type of a C++ polymorphic class type by its vtable and value
+/// represents its counter.
+/// TODO: The class name FunctionId should be renamed to SymbolId in a refactor
+/// change.
+using TypeCountMap = std::map<FunctionId, uint64_t>;
+
+/// Write \p Map to the output stream. Keys are linearized using \p NameTable
+/// and written as ULEB128. Values are written as ULEB128 as well.
+std::error_code
+serializeTypeMap(const TypeCountMap &Map,
+ const MapVector<FunctionId, uint32_t> &NameTable,
+ raw_ostream &OS);
+
/// Representation of a single sample record.
///
/// A sample record is represented by a positive integer value, which
/// indicates how frequently was the associated line location executed.
///
/// Additionally, if the associated location contains a function call,
-/// the record will hold a list of all the possible called targets. For
-/// direct calls, this will be the exact function being invoked. For
-/// indirect calls (function pointers, virtual table dispatch), this
-/// will be a list of one or more functions.
+/// the record will hold a list of all the possible called targets and the types
+/// for virtual table dispatches. For direct calls, this will be the exact
+/// function being invoked. For indirect calls (function pointers, virtual table
+/// dispatch), this will be a list of one or more functions. For virtual table
+/// dispatches, this record will also hold the type of the object.
class SampleRecord {
public:
using CallTarget = std::pair<FunctionId, uint64_t>;
@@ -746,6 +765,7 @@ using BodySampleMap = std::map<LineLocation, SampleRecord>;
// memory, which is *very* significant for large profiles.
using FunctionSamplesMap = std::map<FunctionId, FunctionSamples>;
using CallsiteSampleMap = std::map<LineLocation, FunctionSamplesMap>;
+using CallsiteTypeMap = std::map<LineLocation, TypeCountMap>;
using LocToLocMap =
std::unordered_map<LineLocation, LineLocation, LineLocationHash>;
@@ -939,6 +959,14 @@ class FunctionSamples {
return &Iter->second;
}
+ /// Returns the TypeCountMap for inlined callsites at the given \p Loc.
+ const TypeCountMap *findCallsiteTypeSamplesAt(const LineLocation &Loc) const {
+ auto Iter = VirtualCallsiteTypeCounts.find(mapIRLocToProfileLoc(Loc));
+ if (Iter == VirtualCallsiteTypeCounts.end())
+ return nullptr;
+ return &Iter->second;
+ }
+
/// Returns a pointer to FunctionSamples at the given callsite location
/// \p Loc with callee \p CalleeName. If no callsite can be found, relax
/// the restriction to return the FunctionSamples at callsite location
@@ -1000,6 +1028,46 @@ class FunctionSamples {
return CallsiteSamples;
}
+ /// Returns vtable access samples for the C++ types collected in this
+ /// function.
+ const CallsiteTypeMap &getCallsiteTypeCounts() const {
+ return VirtualCallsiteTypeCounts;
+ }
+
+ /// Returns the vtable access samples for the C++ types for \p Loc.
+ /// Under the hood, the caller-specified \p Loc will be un-drifted before the
+ /// type sample lookup if possible.
+ TypeCountMap &getTypeSamplesAt(const LineLocation &Loc) {
+ return VirtualCallsiteTypeCounts[mapIRLocToProfileLoc(Loc)];
+ }
+
+ /// Scale \p Other sample counts by \p Weight and add the scaled result to the
+ /// type samples for \p Loc. Under the hoold, the caller-provided \p Loc will
+ /// be un-drifted before the type sample lookup if possible.
+ /// typename T is either a std::map or a DenseMap.
+ template <typename T>
+ sampleprof_error addCallsiteVTableTypeProfAt(const LineLocation &Loc,
+ const T &Other,
+ uint64_t Weight = 1) {
+ static_assert((std::is_same_v<typename T::key_type, StringRef> ||
+ std::is_same_v<typename T::key_type, FunctionId>) &&
+ std::is_same_v<typename T::mapped_type, uint64_t>,
+ "T must be a map with StringRef or FunctionId as key and "
+ "uint64_t as value");
+ TypeCountMap &TypeCounts = getTypeSamplesAt(Loc);
+ bool Overflowed = false;
+
+ for (const auto [Type, Count] : Other) {
+ FunctionId TypeId(Type);
+ bool RowOverflow = false;
+ TypeCounts[TypeId] = SaturatingMultiplyAdd(
+ Count, Weight, TypeCounts[TypeId], &RowOverflow);
+ Overflowed |= RowOverflow;
+ }
+ return Overflowed ? sampleprof_error::counter_overflow
+ : sampleprof_error::success;
+ }
+
/// Return the maximum of sample counts in a function body. When SkipCallSite
/// is false, which is the default, the return count includes samples in the
/// inlined functions. When SkipCallSite is true, the return count only
@@ -1054,6 +1122,10 @@ class FunctionSamples {
mergeSampleProfErrors(Result,
FSMap[Rec.first].merge(Rec.second, Weight));
}
+ for (const auto &[Loc, OtherTypeMap] : Other.getCallsiteTypeCounts())
+ mergeSampleProfErrors(
+ Result, addCallsiteVTableTypeProfAt(Loc, OtherTypeMap, Weight));
+
return Result;
}
@@ -1297,6 +1369,23 @@ class FunctionSamples {
/// collected in the call to baz() at line offset 8.
CallsiteSampleMap CallsiteSamples;
+ /// Map a virtual callsite to the list of accessed vtables and vtable counts.
+ /// The callsite is referenced by its source location.
+ ///
+ /// For example, given:
+ ///
+ /// void foo() {
+ /// ...
+ /// 5 inlined_vcall_bar();
+ /// ...
+ /// 5 inlined_vcall_baz();
+ /// ...
+ /// 200 inlined_vcall_qux();
+ /// }
+ /// This map will contain two entries. One with two types for line offset 5
+ /// and one with one type for line offset 200.
+ CallsiteTypeMap VirtualCallsiteTypeCounts;
+
/// IR to profile location map generated by stale profile matching.
///
/// Each entry is a mapping from the location on current build to the matched
diff --git a/llvm/include/llvm/ProfileData/SampleProfReader.h b/llvm/include/llvm/ProfileData/SampleProfReader.h
index bfe079fbe536f..799938ab901c1 100644
--- a/llvm/include/llvm/ProfileData/SampleProfReader.h
+++ b/llvm/include/llvm/ProfileData/SampleProfReader.h
@@ -589,6 +589,10 @@ class SampleProfileReader {
/// Whether the function profiles use FS discriminators.
bool ProfileIsFS = false;
+ /// If true, the profile has vtable profiles and reader should decode them
+ /// to parse profiles correctly.
+ bool ReadVTableProf = false;
+
/// \brief The format of sample.
SampleProfileFormat Format = SPF_None;
@@ -703,6 +707,14 @@ class LLVM_ABI SampleProfileReaderBinary : public SampleProfileReader {
/// otherwise same as readStringFromTable, also return its hash value.
ErrorOr<std::pair<SampleContext, uint64_t>> readSampleContextFromTable();
+ /// Read all virtual functions' vtable access counts for \p FProfile.
+ std::error_code readCallsiteVTableProf(FunctionSamples &FProfile);
+
+ /// Read bytes from the input buffer pointed by `Data` and decode them into
+ /// \p M. `Data` will be advanced to the end of the read bytes when this
+ /// function returns. Returns error if any.
+ std::error_code readVTableTypeCountMap(TypeCountMap &M);
+
/// Points to the current location in the buffer.
const uint8_t *Data = nullptr;
diff --git a/llvm/include/llvm/ProfileData/SampleProfWriter.h b/llvm/include/llvm/ProfileData/SampleProfWriter.h
index e84b2095efd7b..9dbeaf56509b0 100644
--- a/llvm/include/llvm/ProfileData/SampleProfWriter.h
+++ b/llvm/include/llvm/ProfileData/SampleProfWriter.h
@@ -217,13 +217,20 @@ class LLVM_ABI SampleProfileWriterBinary : public SampleProfileWriter {
std::error_code writeBody(const FunctionSamples &S);
inline void stablizeNameTable(MapVector<FunctionId, uint32_t> &NameTable,
std::set<FunctionId> &V);
-
+
MapVector<FunctionId, uint32_t> NameTable;
-
+
void addName(FunctionId FName);
virtual void addContext(const SampleContext &Context);
void addNames(const FunctionSamples &S);
+ /// Write \p CallsiteTypeMap to the output stream \p OS.
+ std::error_code
+ writeCallsiteVTableProf(const CallsiteTypeMap &CallsiteTypeMap,
+ raw_ostream &OS);
+
+ bool WriteVTableProf = false;
+
private:
LLVM_ABI friend ErrorOr<std::unique_ptr<SampleProfileWriter>>
SampleProfileWriter::create(std::unique_ptr<raw_ostream> &OS,
@@ -412,8 +419,7 @@ class LLVM_ABI SampleProfileWriterExtBinaryBase
class LLVM_ABI SampleProfileWriterExtBinary
: public SampleProfileWriterExtBinaryBase {
public:
- SampleProfileWriterExtBinary(std::unique_ptr<raw_ostream> &OS)
- : SampleProfileWriterExtBinaryBase(OS) {}
+ SampleProfileWriterExtBinary(std::unique_ptr<raw_ostream> &OS);
private:
std::error_code writeDefaultLayout(const SampleProfileMap &ProfileMap);
diff --git a/llvm/lib/ProfileData/SampleProf.cpp b/llvm/lib/ProfileData/SampleProf.cpp
index 60c1393616713..ac7513ef2cb49 100644
--- a/llvm/lib/ProfileData/SampleProf.cpp
+++ b/llvm/lib/ProfileData/SampleProf.cpp
@@ -47,6 +47,24 @@ bool FunctionSamples::ProfileIsPreInlined = false;
bool FunctionSamples::UseMD5 = false;
bool FunctionSamples::HasUniqSuffix = true;
bool FunctionSamples::ProfileIsFS = false;
+
+std::error_code
+serializeTypeMap(const TypeCountMap &Map,
+ const MapVector<FunctionId, uint32_t> &NameTable,
+ raw_ostream &OS) {
+ encodeULEB128(Map.size(), OS);
+ for (const auto &[TypeName, SampleCount] : Map) {
+ if (auto NameIndexIter = NameTable.find(TypeName);
+ NameIndexIter != NameTable.end()) {
+ encodeULEB128(NameIndexIter->second, OS);
+ } else {
+ // If the type is not in the name table, we cannot serialize it.
+ return sampleprof_error::truncated_name_table;
+ }
+ encodeULEB128(SampleCount, OS);
+ }
+ return sampleprof_error::success;
+}
} // namespace sampleprof
} // namespace llvm
@@ -178,6 +196,17 @@ raw_ostream &llvm::sampleprof::operator<<(raw_ostream &OS,
return OS;
}
+static void printTypeCountMap(raw_ostream &OS, LineLocation Loc,
+ const TypeCountMap &TypeCountMap) {
+ if (TypeCountMap.empty()) {
+ return;
+ }
+ OS << Loc << ": vtables: ";
+ for (const auto &[Type, Count] : TypeCountMap)
+ OS << Type << ":" << Count << " ";
+ OS << "\n";
+}
+
/// Print the samples collected for a function on stream \p OS.
void FunctionSamples::print(raw_ostream &OS, unsigned Indent) const {
if (getFunctionHash())
@@ -192,7 +221,13 @@ void FunctionSamples::print(raw_ostream &OS, unsigned Indent) const {
SampleSorter<LineLocation, SampleRecord> SortedBodySamples(BodySamples);
for (const auto &SI : SortedBodySamples.get()) {
OS.indent(Indent + 2);
+ const auto &Loc = SI->first;
OS << SI->first << ": " << SI->second;
+ if (const TypeCountMap *TypeCountMap =
+ this->findCallsiteTypeSamplesAt(Loc)) {
+ OS.indent(Indent + 2);
+ printTypeCountMap(OS, Loc, *TypeCountMap);
+ }
}
OS.indent(Indent);
OS << "}\n";
@@ -214,6 +249,11 @@ void FunctionSamples::print(raw_ostream &OS, unsigned Indent) const {
OS << Loc << ": inlined callee: " << FuncSample.getFunction() << ": ";
FuncSample.print(OS, Indent + 4);
}
+ auto TypeSamplesIter = VirtualCallsiteTypeCounts.find(Loc);
+ if (TypeSamplesIter != VirtualCallsiteTypeCounts.end()) {
+ OS.indent(Indent + 2);
+ printTypeCountMap(OS, Loc, TypeSamplesIter->second);
+ }
}
OS.indent(Indent);
OS << "}\n";
diff --git a/llvm/lib/ProfileData/SampleProfReader.cpp b/llvm/lib/ProfileData/SampleProfReader.cpp
index 12769a391286c..81ae792e70b99 100644
--- a/llvm/lib/ProfileData/SampleProfReader.cpp
+++ b/llvm/lib/ProfileData/SampleProfReader.cpp
@@ -197,8 +197,37 @@ enum class LineType {
CallSiteProfile,
BodyProfile,
Metadata,
+ VirtualCallTypeProfile,
};
+// Parse `Input` as a white-space separated list of `vtable:count` pairs. An
+// example input line is `_ZTVbar:1471 _ZTVfoo:630`.
+static bool parseTypeCountMap(StringRef Input,
+ DenseMap<StringRef, uint64_t> &TypeCountMap) {
+ for (size_t Index = Input.find_first_not_of(' '); Index != StringRef::npos;) {
+ size_t ColonIndex = Input.find(':', Index);
+ if (ColonIndex == StringRef::npos)
+ return false; // No colon found, invalid format.
+ StringRef TypeName = Input.substr(Index, ColonIndex - Index);
+ // CountIndex is the start index of count.
+ size_t CountStartIndex = ColonIndex + 1;
+ // NextIndex is the start index after the 'target:count' pair.
+ size_t NextIndex = Input.find_first_of(' ', CountStartIndex);
+ uint64_t Count;
+ if (Input.substr(CountStartIndex, NextIndex - CountStartIndex)
+ .getAsInteger(10, Count))
+ return false; // Invalid count.
+ // Error on duplicated type names in one line of input.
+ auto [Iter, Inserted] = TypeCountMap.insert({TypeName, Count});
+ if (!Inserted)
+ return false;
+ Index = (NextIndex == StringRef::npos)
+ ? StringRef::npos
+ : Input.find_first_not_of(' ', NextIndex);
+ }
+ return true;
+}
+
/// Parse \p Input as line sample.
///
/// \param Input input line.
@@ -215,6 +244,7 @@ static bool ParseLine(const StringRef &Input, LineType &LineTy, uint32_t &Depth,
uint64_t &NumSamples, uint32_t &LineOffset,
uint32_t &Discriminator, StringRef &CalleeName,
DenseMap<StringRef, uint64_t> &TargetCountMap,
+ DenseMap<StringRef, uint64_t> &TypeCountMap,
uint64_t &FunctionHash, uint32_t &Attributes,
bool &IsFlat) {
for (Depth = 0; Input[Depth] == ' '; Depth++)
@@ -306,6 +336,10 @@ static bool ParseLine(const StringRef &Input, LineType &LineTy, uint32_t &Depth,
// Change n3 to the next blank space after colon + integer pair.
n3 = n4;
}
+ } else if (Rest.starts_with(kVTableProfPrefix)) {
+ LineTy = LineType::VirtualCallTypeProfile;
+ return parseTypeCountMap(Rest.substr(strlen(kVTableProfPrefix)),
+ TypeCountMap);
} else {
LineTy = LineType::CallSiteProfile;
size_t n3 = Rest.find_last_of(':');
@@ -374,19 +408,27 @@ std::error_code SampleProfileReaderText::readImpl() {
uint64_t NumSamples;
StringRef FName;
DenseMap<StringRef, uint64_t> TargetCountMap;
+ DenseMap<StringRef, uint64_t> TypeCountMap;
uint32_t Depth, LineOffset, Discriminator;
LineType LineTy = LineType::BodyProfile;
uint64_t FunctionHash = 0;
uint32_t Attributes = 0;
bool IsFlat = false;
+ // TODO: Update ParseLine to return an error code instead of a bool and
+ // report it.
if (!ParseLine(*LineIt, LineTy, Depth, NumSamples, LineOffset,
- Discriminator, FName, TargetCountMap, FunctionHash,
- Attributes, IsFlat)) {
+ Discriminator, FName, TargetCountMap, TypeCountMap,
+ FunctionHash, Attributes, IsFlat)) {
switch (LineTy) {
case LineType::Metadata:
reportError(LineIt.line_number(),
"Cannot parse metadata: " + *LineIt);
break;
+ case LineType::VirtualCallTypeProfile:
+ reportError(LineIt.line_number(),
+ "Expected 'vtables [mangled_vtable:NUM]+', found " +
+ *LineIt);
+ break;
default:
reportError(LineIt.line_number(),
"Expected 'NUM[.NUM]: NUM[ mangled_name:NUM]*', found " +
@@ -417,6 +459,14 @@ std::error_code SampleProfileReaderText::readImpl() {
DepthMetadata = 0;
break;
}
+
+ case LineType::VirtualCallTypeProfile: {
+ mergeSampleProfErrors(
+ Result, InlineStack.back()->addCallsiteVTableTypeProfAt(
+ LineLocation(LineOffset, Discriminator), TypeCountMap));
+ break;
+ }
+
case LineType::BodyProfile: {
FunctionSamples &FProfile = *InlineStack.back();
for (const auto &name_count : TargetCountMap) {
@@ -598,6 +648,67 @@ SampleProfileReaderBinary::readSampleContextFromTable() {
return std::make_pair(Context, Hash);
}
+std::error_code
+SampleProfileReaderBinary::readVTableTypeCountMap(TypeCountMap &M) {
+ auto NumVTableTypes = readNumber<uint32_t>();
+ if (std::error_code EC = NumVTableTypes.getError())
+ return EC;
+
+ for (uint32_t I = 0; I < *NumVTableTypes; ++I) {
+ auto VTableType(readStringFromTable());
+ if (std::error_code EC = VTableType.getError())
+ return EC;
+
+ auto VTableSamples = readNumber<uint64_t>();
+ if (std::error_code EC = VTableSamples.getError())
+ return EC;
+ // The source profile should not have duplicate vtable records at the same
+ // location. In case duplicate vtables are found, reader can emit a warning
+ // but continue processing the profile.
+ if (!M.insert(std::make_pair(*VTableType, *VTableSamples)).second) {
+ Ctx.diagnose(DiagnosticInfoSampleProfile(
+ Buffer->getBufferIdentifier(), 0,
+ "Duplicate vtable type " + VTableType->str() +
+ " at the same location. Additional counters will be ignored.",
+ DS_Warning));
+ continue;
+ }
+ }
+ return sampleprof_error::success;
+}
+
+std::error_code
+SampleProfileReaderBinary::readCallsiteVTableProf(FunctionSamples &FProfile) {
+ assert(ReadVTableProf &&
+ "Cannot read vtable profiles if ReadVTableProf is false");
+
+ // Read the vtable type profile for the callsite.
+ auto NumCallsites = readNumber<uint32_t>();
+ if (std::error_code EC = NumCallsites.getError())
+ return EC;
+
+ for (uint32_t I = 0; I < *NumCallsites; ++I) {
+ auto LineOffset = readNumber<uint64_t>();
+ if (std::error_code EC = LineOffset.getError())
+ return EC;
+
+ if (!isOffsetLegal(*LineOffset))
+ return sampleprof_error::illegal_line_offset;
+
+ auto Discriminator = readNumber<uint64_t>();
+ if (std::error_code EC = Discriminator.getError())
+ return EC;
+
+ // Here we handle FS discriminators:
+ const uint32_t DiscriminatorVal = (*Discriminator) & getDiscriminatorMask();
+
+ if (std::error_code EC = readVTableTypeCountMap(FProfile.getTypeSamplesAt(
+ LineLocation(*LineOffset, DiscriminatorVal))))
+ return EC;
+ }
+ return sampleprof_error::success;
+}
+
std::error_code
SampleProfileReaderBinary::readProfile(FunctionSamples &FProfile) {
auto NumSamples = readNumber<uint64_t>();
@@ -678,6 +789,9 @@ SampleProfileReaderBinary::readProfile(FunctionSamples &FProfile) {
return EC;
}
+ if (ReadVTableProf)
+ return readCallsiteVTableProf(FProfile);
+
return sampleprof_error::success;
}
@@ -740,6 +854,8 @@ std::error_code SampleProfileReaderExtBinaryBase::readOneSection(
FunctionSamples::ProfileIsPreInlined = ProfileIsPreInlined = true;
if (hasSecFlag(Entry, SecProfSummaryFlags::SecFlagFSDiscriminator))
FunctionSamples::ProfileIsFS = ProfileIsFS = true;
+ if (hasSecFlag(Entry, SecProfSummaryFlags::SecFlagHasVTableTypeProf))
+ ReadVTableProf = true;
break;
case SecNameTable: {
bool FixedLengthMD5 =
diff --git a/llvm/lib/ProfileData/SampleProfWriter.cpp b/llvm/lib/ProfileData/SampleProfWriter.cpp
index 9173a0f94f69d..e5f31348578b8 100644
--- a/llvm/lib/ProfileData/SampleProfWriter.cpp
+++ b/llvm/lib/ProfileData/SampleProfWriter.cpp
@@ -41,6 +41,11 @@
using namespace llvm;
using namespace sampleprof;
+// To begin with, make this option off by default.
+static cl::opt<bool> ExtBinaryWriteVTableTypeProf(
+ "extbinary-write-vtable-type-prof", cl::init(false), cl::Hidden,
+ cl::desc("Write vtable type profile in ext-binary sample profile writer"));
+
namespace llvm {
namespace support {
namespace endian {
@@ -435,6 +440,9 @@ std::error_code SampleProfileWriterExtBinaryBase::writeOneSection(
addSectionFlag(SecProfSummary, SecProfSummaryFlags::SecFlagIsPreInlined);
if (Type == SecProfSummary && FunctionSamples::ProfileIsFS)
addSectionFlag(SecProfSummary, SecProfSummaryFlags::SecFlagFSDiscriminator);
+ if (Type == SecProfSummary && ExtBinaryWriteVTableTypeProf)
+ addSectionFlag(SecProfSummary,
+ SecProfSummaryFlags::SecFlagHasVTableTypeProf);
uint64_t SectionStart = markSectionStart(Type, LayoutIdx);
switch (Type) {
@@ -478,6 +486,12 @@ std::error_code SampleProfileWriterExtBinaryBase::writeOneSection(
return sampleprof_error::success;
}
+SampleProfileWriterExtBinary::SampleProfileWriterExtBinary(
+ std::unique_ptr<raw_ostream> &OS)
+ : SampleProfileWriterExtBinaryBase(OS) {
+ WriteVTableProf = ExtBinaryWriteVTableTypeProf;
+}
+
std::error_code SampleProfileWriterExtBinary::writeDefaultLayout(
const SampleProfileMap &ProfileMap) {
// The const indices passed to writeOneSection below are specifying the
@@ -587,6 +601,19 @@ std::error_code SampleProfileWriterText::writeSample(const FunctionSamples &S) {
OS << " " << J.first << ":" << J.second;
OS << "\n";
LineCount++;
+
+ if (const TypeCountMap *Map = S.findCallsiteTypeSamplesAt(Loc);
+ Map && !Map->empty()) {
+ OS.indent(Indent + 1);
+ Loc.print(OS);
+ OS << ": ";
+ OS << kVTableProfPrefix;
+ for (const auto [TypeName, Count] : *Map) {
+ OS << TypeName << ":" << Count << " ";
+ }
+ OS << "\n";
+ LineCount++;
+ }
}
SampleSorter<LineLocation, FunctionSamplesMap> SortedCallsiteSamples(
@@ -603,7 +630,21 @@ std::error_code SampleProfileWriterText::writeSample(const FunctionSamples &S) {
if (std::error_code EC = writeSample(CalleeSamples))
return EC;
}
+
+ if (const TypeCountMap *Map = S.findCallsiteTypeSamplesAt(Loc);
+ Map && !Map->empty()) {
+ OS.indent(Indent);
+ Loc.print(OS);
+ OS << ": ";
+ OS << kVTableProfPrefix;
+ for (const auto [TypeId, Count] : *Map) {
+ OS << TypeId << ":" << Count << " ";
+ }
+ OS << "\n";
+ LineCount++;
+ }
}
+
Indent -= 1;
if (FunctionSamples::ProfileIsProbeBased) {
@@ -663,6 +704,17 @@ void SampleProfileWriterBinary::addNames(const FunctionSamples &S) {
addName(CalleeSamples.getFunction());
addNames(CalleeSamples);
}
+
+ if (!WriteVTableProf)
+ return;
+ // Add all the vtable names to NameTable.
+ for (const auto &VTableAccessCountMap :
+ llvm::make_second_range(S.getCallsiteTypeCounts())) {
+ // Add type name to NameTable.
+ for (const auto Type : llvm::make_first_range(VTableAccessCountMap)) {
+ addName(Type);
+ }
+ }
}
void SampleProfileWriterExtBinaryBase::addContext(
@@ -801,6 +853,22 @@ std::error_code SampleProfileWriterExtBinaryBase::writeHeader(
return sampleprof_error::success;
}
+std::error_code SampleProfileWriterBinary::writeCallsiteVTableProf(
+ const CallsiteTypeMap &CallsiteTypeMap, raw_ostream &OS) {
+ assert(WriteVTableProf &&
+ "writeCallsiteVTableProf should not be called if WriteVTableProf is "
+ "false");
+
+ encodeULEB128(CallsiteTypeMap.size(), OS);
+ for (const auto &[Loc, TypeMap] : CallsiteTypeMap) {
+ Loc.serialize(OS);
+ if (std::error_code EC = serializeTypeMap(TypeMap, getNameTable(), OS))
+ return EC;
+ }
+
+ return sampleprof_error::success;
+}
+
std::error_code SampleProfileWriterBinary::writeSummary() {
auto &OS = *OutputStream;
encodeULEB128(Summary->getTotalCount(), OS);
@@ -838,14 +906,16 @@ std::error_code SampleProfileWriterBinary::writeBody(const FunctionSamples &S) {
for (const auto &J : S.getCallsiteSamples())
NumCallsites += J.second.size();
encodeULEB128(NumCallsites, OS);
- for (const auto &[Loc, CalleeFunctionSampleMap] : S.getCallsiteSamples())
- for (const auto &FunctionSample :
- llvm::make_second_range(CalleeFunctionSampleMap)) {
- Loc.serialize(OS);
- if (std::error_code EC = writeBody(FunctionSample))
+ for (const auto &J : S.getCallsiteSamples())
+ for (const auto &FS : J.second) {
+ J.first.serialize(OS);
+ if (std::error_code EC = writeBody(FS.second))
return EC;
}
+ if (WriteVTableProf)
+ return writeCallsiteVTableProf(S.getCallsiteTypeCounts(), OS);
+
return sampleprof_error::success;
}
diff --git a/llvm/test/tools/llvm-profdata/Inputs/profile-symbol-list-ext.expected b/llvm/test/tools/llvm-profdata/Inputs/profile-symbol-list-ext.expected
new file mode 100644
index 0000000000000..f7e7499a2c781
--- /dev/null
+++ b/llvm/test/tools/llvm-profdata/Inputs/profile-symbol-list-ext.expected
@@ -0,0 +1,44 @@
+Function: main: 368038, 0, 7 sampled lines
+Samples collected in the function's body {
+ 4: 1068
+ 4.2: 1068
+ 5: 2150
+ 5.1: 2150
+ 6: 4160
+ 7: 1068
+ 9: 4128, calls: _Z3bari:2942 _Z3fooi:1262
+ 9: vtables: _ZTVbar:2942 _ZTVfoo:1260
+}
+Samples collected in inlined callsites {
+ 10: inlined callee: inline1: 2000, 0, 1 sampled lines
+ Samples collected in the function's body {
+ 1: 2000
+ }
+ No inlined callsites in this function
+ 10: inlined callee: inline2: 4000, 0, 1 sampled lines
+ Samples collected in the function's body {
+ 1: 4000
+ }
+ No inlined callsites in this function
+ 10: vtables: _ZTVinline1:2000 _ZTVinline2:4000
+}
+Function: _Z3bari: 40602, 2874, 1 sampled lines
+Samples collected in the function's body {
+ 1: 2874
+}
+No inlined callsites in this function
+Function: _Z3fooi: 15422, 1220, 1 sampled lines
+Samples collected in the function's body {
+ 1: 1220
+}
+No inlined callsites in this function
+======== Dump profile symbol list ========
+_Z3goov
+_Z3sumii
+__libc_csu_fini
+__libc_csu_init
+_dl_relocate_static_pie
+_fini
+_init
+_start
+main
diff --git a/llvm/test/tools/llvm-profdata/Inputs/sample-profile-ext.proftext b/llvm/test/tools/llvm-profdata/Inputs/sample-profile-ext.proftext
new file mode 100644
index 0000000000000..100133fa17ccb
--- /dev/null
+++ b/llvm/test/tools/llvm-profdata/Inputs/sample-profile-ext.proftext
@@ -0,0 +1,18 @@
+main:184019:0
+ 4: 534
+ 4.2: 534
+ 5: 1075
+ 5.1: 1075
+ 6: 2080
+ 7: 534
+ 9: 2064 _Z3bari:1471 _Z3fooi:631
+ 9: vtables _ZTVbar:1471 _ZTVfoo:630
+ 10: inline1:1000
+ 1: 1000
+ 10: inline2:2000
+ 1: 2000
+ 10: vtables _ZTVinline1:1000 _ZTVinline2:2000
+_Z3bari:20301:1437
+ 1: 1437
+_Z3fooi:7711:610
+ 1: 610
diff --git a/llvm/test/tools/llvm-profdata/profile-symbol-list-compress.test b/llvm/test/tools/llvm-profdata/profile-symbol-list-compress.test
index b445695c8e8e4..8383bcc1a2fbe 100644
--- a/llvm/test/tools/llvm-profdata/profile-symbol-list-compress.test
+++ b/llvm/test/tools/llvm-profdata/profile-symbol-list-compress.test
@@ -4,3 +4,12 @@ REQUIRES: zlib
; RUN: llvm-profdata merge -sample -extbinary -compress-all-sections %t.1.output %t.2.output -o %t.3.output
; RUN: llvm-profdata show -sample -show-prof-sym-list %t.3.output > %t.4.output
; RUN:
diff -b %S/Inputs/profile-symbol-list.expected %t.4.output
+
+;; Generate two SampleFDO binary profiles and merge them.
+;; Tests that the vtable counters in the merged profile are the aggregated
+;; result from both sources.
+; RUN: llvm-profdata merge -sample -extbinary -compress-all-sections -extbinary-write-vtable-type-prof -prof-sym-list=%S/Inputs/profile-symbol-list-1.text %S/Inputs/sample-profile-ext.proftext -o %t.1.output
+; RUN: llvm-profdata merge -sample -extbinary -compress-all-sections -extbinary-write-vtable-type-prof -prof-sym-list=%S/Inputs/profile-symbol-list-2.text %S/Inputs/sample-profile-ext.proftext -o %t.2.output
+; RUN: llvm-profdata merge -sample -extbinary -compress-all-sections -extbinary-write-vtable-type-prof %t.1.output %t.2.output -o %t.3.output
+; RUN: llvm-profdata show -sample -show-prof-sym-list %t.3.output > %t.4.output
+; RUN:
diff -b %S/Inputs/profile-symbol-list-ext.expected %t.4.output
diff --git a/llvm/test/tools/llvm-profdata/profile-symbol-list.test b/llvm/test/tools/llvm-profdata/profile-symbol-list.test
index 39dcd11ec1db7..6845531066c76 100644
--- a/llvm/test/tools/llvm-profdata/profile-symbol-list.test
+++ b/llvm/test/tools/llvm-profdata/profile-symbol-list.test
@@ -7,3 +7,12 @@
; RUN: llvm-profdata show -sample -show-sec-info-only %t.5.output | FileCheck %s -check-prefix=NOSYMLIST
; NOSYMLIST: ProfileSymbolListSection {{.*}} Size: 0
+
+;; Generate two SampleFDO binary profiles and merge them.
+;; Tests that the vtable counters in the merged profile are the aggregated
+;; result from both sources.
+; RUN: llvm-profdata merge -sample -extbinary -extbinary-write-vtable-type-prof -prof-sym-list=%S/Inputs/profile-symbol-list-1.text %S/Inputs/sample-profile-ext.proftext -o %t.1.output
+; RUN: llvm-profdata merge -sample -extbinary -extbinary-write-vtable-type-prof -prof-sym-list=%S/Inputs/profile-symbol-list-2.text %S/Inputs/sample-profile-ext.proftext -o %t.2.output
+; RUN: llvm-profdata merge -sample -extbinary -extbinary-write-vtable-type-prof %t.1.output %t.2.output -o %t.3.output
+; RUN: llvm-profdata show -sample -show-prof-sym-list %t.3.output > %t.4.output
+; RUN:
diff -b %S/Inputs/profile-symbol-list-ext.expected %t.4.output
diff --git a/llvm/test/tools/llvm-profdata/roundtrip.test b/llvm/test/tools/llvm-profdata/roundtrip.test
index 7af76e0a58224..eb55534763877 100644
--- a/llvm/test/tools/llvm-profdata/roundtrip.test
+++ b/llvm/test/tools/llvm-profdata/roundtrip.test
@@ -16,3 +16,9 @@ RUN: llvm-profdata merge --sample --binary -output=%t.4.profdata %S/Inputs/sampl
RUN: llvm-profdata merge --sample --extbinary -output=%t.5.profdata %t.4.profdata
RUN: llvm-profdata merge --sample --text -output=%t.4.proftext %t.5.profdata
RUN:
diff -b %t.4.proftext %S/Inputs/sample-profile.proftext
+# Round trip from text --> extbinary --> text.
+# The vtable profile is supported by ext-binary profile but not raw binary profile format,
+# so we don't use raw binary profile format in this roundtrip.
+RUN: llvm-profdata merge --sample --extbinary -extbinary-write-vtable-type-prof --output=%t.5.profdata %S/Inputs/sample-profile-ext.proftext
+RUN: llvm-profdata merge --sample --text --output=%t.5.proftext %t.5.profdata
+RUN:
diff -b %t.5.proftext %S/Inputs/sample-profile-ext.proftext
More information about the llvm-commits
mailing list