[LLVMdev] RFC: Binary format for instrumentation based profiling data

[Justin Bogner]

Chandler Carruth <chandlerc at google.com> writes:
> I think it would be worthwhile to consider the alternative of having
> the profile library write out data files in a format which is
> essentially "always" transformed by a post-processing tool before
> being used during compilation.

We seem to have some agreement that two formats for instrumentation
based profiling is worthwhile. These are that emitted by compiler-rt in
the instrumented program at runtime (format 1), and that which is
consumed by clang when compiling the program with PGO (format 2).

Format 1
--------

This format should be efficient to write, since the instrumented program
should run with as little overhead as possible. This also doesn't need
to be stable, and we can assume the same version of LLVM that was used
to instrument the program will read the counter data. As such, the file
format is versioned (so we can easily reject versions we don't
understand) and consists basically of a memory dump of the relevant
profiling counters.

Format 2
--------

This format should be efficient to read and preferably reasonably
compact. We'll convert from format 1 to format 2 using llvm-profdata,
and clang will use format 2 for PGO.

Since the only particularly important operation in this use case is fast
lookup, I propose using the on disk hash table that's currently used in
clang for AST serialization/PTH/etc with a small amount of metadata in a
header.

The hash table implementation currently lives in include/clang/Basic and
consists of a single header. Moving it to llvm and updating the clients
in clang should be easy. I'll send a brief RFC separately to see if
anyone's opposed to moving it.


Thoughts?