[compiler-rt] [llvm] [docs][IRPGO]Document two binary formats for IRPGO profiles (PR #76105)
David Li via llvm-commits
llvm-commits at lists.llvm.org
Thu Dec 21 11:01:58 PST 2023
================
@@ -0,0 +1,395 @@
+===================================
+Instrumentation PGO Profile Format
+===================================
+
+.. contents::
+ :local:
+
+
+Overview
+=========
+
+Instrumentation PGO inserts `llvm.instrprof.*` `code generator intrinsics`_
+in the code to generate profiles. This document describes two binary profile
+formats (raw and indexed) used by instrumentation.
+
+.. _`code generator intrinsics`: https://llvm.org/docs/LangRef.html#code-generator-intrinsics
+
+.. note::
+ The instrumentation profile format supports non-PGO use cases (e.g., temporal
+ profiling). This document will focus on PGO. Source coverage uses both
+ frontend instrumentation profiles and coverage mapping. The format for
+ coverage mapping has its own `documentation`_.
+
+.. _`documentation`: https://llvm.org/docs/CoverageMappingFormat.html
+
+Raw Profile Format
+===================
+
+The raw profile is generated by running the instrumented binary. It is a memory
+dump of the profile data.
+
+The instrumented binary currently collects two kinds of profile data, counters
+to profile branch probability and (various flavors of) value profiles. The
+profile data for a function span across several sections in the profile.
+
+General Storage Layout
+-----------------------
+
+A raw profile from an executable or a shared library [1]_ consists of a profile
+header and several sections. The storage layout is illustrated below. Generally,
+when the raw profile is read into an memory buffer, the actual byte offset of a
+section is inferred from the section's order in the layout and size information
+of all the sections ahead of it.
+
+::
+
+ +----+-----------------------+
+ | | Magic |
+ | +-----------------------+
+ | | Version |
+ | +-----------------------+
+ H | Size Info for |
+ E | Section 1 |
+ A +-----------------------+
+ D | Size Info for |
+ E | Section 2 |
+ R +-----------------------+
+ | | ... |
+ | +-----------------------+
+ | | Size Info for |
+ | | Section N |
+ +----+-----------------------+
+ P | Section 1 |
+ A +-----------------------+
+ Y | Section 2 |
+ L +-----------------------+
+ O | ... |
+ A +-----------------------+
+ D | Section N |
+ +----+-----------------------+
+
+
+.. note::
+ Sections might be padded to meet platform-specific alignment requirements.
+ For simplicity, header fields and data sections solely for padding purpose
+ are omitted in the data layout graph above and the rest of this document.
+
+Header
+-------
+
+``Magic``
+ With the magic number, data consumer could detect profile format and
+ endianness of the data, and tells whether/how to continue reading.
+
+``Version``
+ The lower 32 bits specifies the actual version and the most significant 32
+ bits specify the variant types of the profile. IR-based instrumentation PGO
+ and context-sensitive IR-based instrumentation PGO are two variant types.
+
+``BinaryIdsSize``
+ The byte size of binary id section.
+
+``NumData``
+ The number of per-function profile data control structures. The byte size of
+ profile data section could be computed with this field.
+
+``NumCounter``
+ The number of entries in the profile counter section. The byte size of counter
+ section could be computed with this field.
+
+``NumBitmapBytes``
+ The number of bytes in the profile bitmap section.
+
+``NamesSize``
+ The number of bytes in the name section.
+
+``CountersDelta``
+ Records the in-memory address difference between the data and counter section,
+ i.e., `start(__llvm_prf_cnts) - start(__llvm_prf_data)`. It's used jointly
+ with the in-memory address difference of profile data record and its counter
+ to find the counter of a profile data record. Check out calculation-of-counter-offset_
+ for details.
+
+``BitmapDelta``
+ Records the in-memory address difference between the data and bitmap section,
+ i.e., `start(__llvm_prf_bits) - start(__llvm_prf_data)`. It's used jointly
+ with the in-memory address difference of a profile data record and its bitmap
+ to find the bitmap of a profile data record, in a similar way to how counters
+ are referenced as explained by calculation-of-counter-offset_ .
+
+``NamesDelta``
+ Records the in-memory address of name section. Not used except for raw profile
+ reader error checking.
+
+``ValueKindLast``
+ Records the number of value kinds. As of writing, two kinds of value profiles
+ are supported. `IndirectCallTarget` is to profile the frequent callees of
+ indirect call instructions and `MemOPSize` is for memory intrinsic function
+ size profiling.
+
+ The number of value kinds affects the byte size of per function profile data
+ control structure.
+
+Payload Sections
+------------------
+
+Binary Ids
+^^^^^^^^^^^
+Stores the binary ids of the instrumented binaries to associate binaries with
+profiles for source code coverage. See `Binary Id RFC`_ for introduction.
+
+.. _`Binary Id RFC`: https://lists.llvm.org/pipermail/llvm-dev/2021-June/151154.html
+
+Profile Data
+^^^^^^^^^^^^^
+
+This section stores per-function profile data control structure. The in-memory
+representation of the control structure is `__llvm_profile_data` and the fields
+are defined by `INSTRPROFDATA` macro. Some fields are used to reference data
+from other sections in the profile. The fields are documented as follows:
+
+``NameRef``
+ The MD5 of the function's PGO name. PGO name has the format
+ `[<filepath><delimiter>]<linkage-or-mangled-name>` where `<filepath>` and
+ `<delimiter>` is provided for local-linkage functions to tell possibly
+ identical functions.
+
+``FuncHash``
+ A fingerprint of the function's control flow graph.
+
+``CounterPtr``
+ The in-memory address difference between profile data and its corresponding
+ counters.
+
+``BitmapPtr``
+ The in-memory address difference between profile data and its bitmap.
+
+``FunctionPointer``
+ Records the function address when instrumented binary runs. This is used to
+ map the profiled callee address of indirect calls to the `NameRef` during
+ conversion from raw to indexed profiles.
+
+``Values``
+ Represents value profiles in a two dimensional array. The number of elements
+ in the first dimension is the number of instrumented value sites across all
+ kinds. Each element in the first dimension is the head of a linked list, and
+ the each element in the second dimension is linked list element, carrying
+ `<profiled-value, count>` as payload. This is used by compiler runtime when
+ writing out value profiles.
+
+``NumCounters``
+ The number of counters for the instrumented function.
+
+``NumValueSites``
+ This is an array of counters, and each counter represents the number of
+ instrumented sites for a kind of value in the function.
+
+``NumBitmapBytes``
+ The number of bitmap bytes for the function.
+
+Profile Counters
+^^^^^^^^^^^^^^^^^
+
+For PGO [2]_, the counters within an instrumented function are stored contiguously
+and in an order that is consistent with instrumentation points selection in the
+instrumentation pass.
+
+.. _calculation-of-counter-offset:
+
+So how are function counters associated with a function?
+
+Basically, the profile reader iterates per-function control structure (from the
+profile data section) and makes use of the recorded relative distances, as
+illustrated below.
+
+::
+
+ + --> start(__llvm_prf_data) --> +---------------------+ ------------+
+ | | Data 1 | |
+ | +---------------------+ =====|| |
+ | | Data 2 | || |
+ | +---------------------+ || |
+ | | ... | || |
+ Counter| +---------------------+ || |
+ Delta | | Data N | || |
+ | +---------------------+ || | CounterPtr1
+ | || |
+ | CounterPtr2 || |
+ | || |
+ | || |
+ + --> start(__llvm_prf_cnts) --> +---------------------+ || |
+ | ... | || |
+ +---------------------+ -----||----+
+ | Counter 1 | ||
+ +---------------------+ ||
+ | ... | ||
+ +---------------------+ =====||
+ | Counter 2 |
+ +---------------------+
+ | ... |
+ +---------------------+
+ | Counter N |
+ +---------------------+
+
+
+In the graph,
+
+* The profile header records `CounterDelta` with the value as `start(__llvm_prf_cnts) - start(__llvm_prf_data)`.
+ We will call it `CounterDeltaInitVal` below for convenience.
+* For each profile data record, `CounterPtrN` is recorded as `start(Counter) - start(ProfileData)`.
+
+Each time the reader advances to the next data record, it updates `CounterDelta`
+to minus the size of one `ProfileData`.
+
+For the counter corresponding to the first data record, the byte offset
+relative to the start of the counter section is calculated as `CounterPtr1 - CounterDeltaInitVal`.
+When profile reader advances to the second data record, note `CounterDelta`
+is updated to `CounterDeltaInitVal - sizeof(ProfileData)`.
+Thus the byte offset relative to the start of the counter section is calculated
+as `CounterPtr2 - (CounterDeltaInitVal - sizeof(ProfileData))`.
+
+Bitmap
+^^^^^^^
+This section is used for source-based MC/DC code coverage. Check out `Bitmap RFC`_
+if interested.
+
+.. _`Bitmap RFC`: https://discourse.llvm.org/t/rfc-source-based-mc-dc-code-coverage/59244
+
+Names
+^^^^^^
+
+This section contains possibly compressed concatenated string of functions' PGO
+names. If compressed, zlib compression algorithm is used.
+
+Function names serve as keys in the PGO data hash table when raw profiles are
+converted into indexed profiles. They are also crucial for `llvm-profdata` to
+show the profiles in a human-readable way.
+
+Value Profile Data
+^^^^^^^^^^^^^^^^^^^^
+
+This section contains the profile data for value profiling.
+
+The value profiles corresponding to a profile data are serialized contiguously
+as one record, and value profile records are stored in the same order as the
+respective profile data, such that a raw profile reader advances the pointer to
+profile data and the pointer to value profile records simutaneously [3]_ to find
+value profiles for a per function, per cfg fingerprint profile data.
+
+Indexed PGO Profile Format
+===========================
+
+General Storage Layout
+-----------------------
+
+::
+
+ +-----------------------+---+
+ | Magic | |
+ +-----------------------+ |
+ | Version | |
+ +-----------------------+ |
+ | HashType | H
+ +-----------------------+ E
+ +-------| HashOffset | A
+ | +-----------------------+ D
+ +-----------| MemProfOffset | E
+ | | +-----------------------+ R
+ | | +--| BinaryIdOffset | |
+ | | | +-----------------------+ |
+ +---------------| TemporalProf- | |
+ | | | | | TracesOffset | |
+ | | | | +-----------------------+---+
+ | | | | | Profile Summary | |
+ | | | | +-----------------------+ P
+ | | +------>| Function PGO data | A
+ | | | +-----------------------+ Y
+ | +---------- | MemProf profile data | L
+ | | +-----------------------+ O
+ | +--| Binary Ids | A
+ | +-----------------------+ D
+ +-------------->| Temporal profiles | |
+ +-----------------------+---+
+
+Header
+--------
+
+``Magic``
+ The purpose of the magic number is to be able to quickly tell if the profile
+ is an indexed profile.
+
+``Version``
+ Similar to raw profile version, the lower 32 bits specifies the version of the
+ indexed profile and the most significant 32 bits are reserved to specify the
+ variant types of the profile.
+
+``HashType``
+ The hashing scheme for on-disk hash table keys. Only MD5 hashing is used as of
+ writing.
+
+``HashOffset``
+ An on-disk hash table stores the per-function profile records. It records the
+ offset of this hash table's metadata (i.e., the number of buckets and entries),
+ which follows right after the payload of the entire hash table for
+ deserialization.
+
+``MemProfOffset``
+ Records the byte offset of MemProf profiling data.
+
+``BinaryIdOffset``
+ Records the byte offset of binary id sections.
+
+``TemporalProfTracesOffset``
+ Records the byte offset of temporal profiles.
+
+Payload Sections
+------------------
+
+(CS) Profile Summary
+^^^^^^^^^^^^^^^^^^^^^
+This section is right after profile header. It stores the serialized profile
+summary. For context-sensitive IR-based instrumentation PGO, this section stores
+an additional profile summary corresponding to the context-sensitive profiles.
+
+Function PGO data
+^^^^^^^^^^^^^^^^^^
+This section stores functions and their PGO profiling data as an on-disk hash
----------------
david-xl wrote:
Profile data for functions with the same name are grouped together and share one hash table entry (the functions may come from different shared libraries for instance). The profile data for them are organized as a sequence of key-value pair where the key is the funcHash (CFG based for IR PGO), and the value is profile counters for the function.
https://github.com/llvm/llvm-project/pull/76105
More information about the llvm-commits
mailing list