[llvm] [compiler-rt] [docs][IRPGO]Document two binary formats for instrumentation-based profiles, with a focus on IRPGO. (PR #76105)

[Fangrui Song via llvm-commits]

================
@@ -0,0 +1,479 @@
+===================================
+Instrumentation Profile Format
+===================================
+
+.. contents::
+   :local:
+
+
+Overview
+=========
+
+Clang supports two types of profiling via instrumentation [1]_: frontend-based
+and IR-based, and both could support a variety of use cases [2]_ .
+This document describes two binary serialization formats (raw and indexed) to
+store instrumented profiles with a specific emphasis on IRPGO use case, in the
+sense that when specific header fields and payload sections have different ways
+of interpretation across use cases, the documentation is based on IRPGO.
+
+.. note::
+  Frontend-generated profiles are used together with coverage mapping for
+  `source based code coverage`_. The `coverage mapping format`_ is different from
+  profile format.
+
+.. _`source based code coverage`: https://clang.llvm.org/docs/SourceBasedCodeCoverage.html
+.. _`coverage mapping format`: https://llvm.org/docs/CoverageMappingFormat.html
+
+Raw Profile Format
+===================
+
+The raw profile is generated by running the instrumented binary. The raw profile
+data from an executable or a shared library [3]_ consists of a header and
+multiple sections, with each section as a memory dump. The profile raw data needs
+to be reasonably compact and fast to generate.
+
+There are no backward or forward version compatiblity guarantees for the raw profile
+format. That is, compilers and tools `require`_ a specific raw profile version
+to parse the profiles.
+
+.. _`require`: https://github.com/llvm/llvm-project/blob/bffdde8b8e5d9a76a47949cd0f574f3ce656e181/llvm/lib/ProfileData/InstrProfReader.cpp#L551-L558
+
+To feed profiles back into compilers for an optimized build (e.g., via
+`-fprofile-use` for IR instrumentation), a raw profile must to be converted into
+indexed format.
+
+General Storage Layout
+-----------------------
+
+The storage layout of raw profile data format is illustrated below. Basically,
+when the raw profile is read into an memory buffer, the actual byte offset of a
+section is inferred from the section's order in the layout and size information
+of all the sections ahead of it.
+
+::
+
+  +----+-----------------------+
+  |    |        Magic          |
+  |    +-----------------------+
+  |    |        Version        |
+  |    +-----------------------+
+  H    |   Size Info for       |
+  E    |      Section 1        |
+  A    +-----------------------+
+  D    |   Size Info for       |
+  E    |      Section 2        |
+  R    +-----------------------+
+  |    |          ...          |
+  |    +-----------------------+
+  |    |   Size Info for       |
+  |    |      Section N        |
+  +----+-----------------------+
+  P    |       Section 1       |
+  A    +-----------------------+
+  Y    |       Section 2       |
+  L    +-----------------------+
+  O    |          ...          |
+  A    +-----------------------+
+  D    |       Section N       |
+  +----+-----------------------+
+
+
+.. note::
+   Sections might be padded to meet specific alignment requirements. For
+   simplicity, header fields and data sections solely for padding purpose are
+   omitted in the data layout graph above and the rest of this document.
+
+Header
+-------
+
+``Magic``
+  Magic number encodes profile format (raw, indexed or text). For the raw format,
+  the magic number also encodes the endianness (big or little) and C pointer
+  byte size (32 or 64) of the platform on which the profile is generated.
+
+  A factory method reads the magic number to construct reader properly and returns
+  error upon unrecognized format. Specifically, the factory method and raw profile
+  reader implementation make sure that a raw profile file could be read back on
+  a platform with the opposite endianness and/or the other C pointer byte size.
+
+``Version``
+  The lower 32 bits specifies the actual version and the most significant 32
+  bits specify the variant types of the profile. IR-based instrumentation PGO
+  and context-sensitive IR-based instrumentation PGO are two variant types.
+
+``BinaryIdsSize``
+  The byte size of `binary id`_ section.
+
+``NumData``
+  The number of profile metadata. The byte size of `profile metadata`_ section
+  could be computed with this field.
+
+``NumCounter``
+  The number of entries in the profile counter section. The byte size of `counter`_
+  section could be computed with this field.
+
+``NumBitmapBytes``
+  The number of bytes in the profile `bitmap`_ section.
+
+``NamesSize``
+  The number of bytes in the name section.
+
+.. _`CountersDelta`:
+
+``CountersDelta``
+  This field records the in-memory address difference between the `profile metadata`_
+  and counter section in the instrumented binary, i.e., `start(__llvm_prf_cnts) - start(__llvm_prf_data)`.
+
+  It's used jointly with the `CounterPtr`_ field to compute the counter offset
+  relative to `start(__llvm_prf_cnts)`. Check out calculation-of-counter-offset_
----------------
MaskRay wrote:

double backticks

https://github.com/llvm/llvm-project/pull/76105