[llvm-dev] InstrProf backward compatibility

[Xinliang David Li via llvm-dev]

On Fri, Dec 18, 2015 at 5:37 AM, Johan Engelen <jbc.engelen at gmail.com> wrote:
> Hi all,
>   I am working on adding PGO to LDC (LLVM D Compiler). The current
> implementation
> 1) uses LLVM's InstrProf pass to generate an instrumented executable
> 2) links to compiler-rt/lib/profile for the runtime functionality to write a
> raw profile data file
> 3) uses llvm-profdata to merge profile data and convert from profraw to
> profdata format
> 4) uses llvm::IndexedInstrProfReader to read-in the profdata file for the
> 2nd PGO compile pass
>
> This all works well with LLVM 3.7. While testing this with LLVM trunk, I run
> into a version compatibility problem between different versions of LLVM: I
> made a local copy of the compiler-rt/lib/profile release_37 version (with a
> few fixes for MSVC) and so steps 1,2, and 4 use LLVM trunk, but step 2 uses
> an older version of compiler-rt/lib/profile. Apparently, trunk's

Runtime and InstrProf pass have to be in sync. You can not use 3.7
version of runtime with trunk version of 1).

Perhaps you want to explain more about your use cases? If you want to
use 3.7 compiler/runtime to produce profile to be used by trunk
compiler -- that is doable.

1) instrument the code with 3.7 compiler and link with 3.7 runtime
2) run and dump the profile
3) use 3.7 llvm-profdata tool to do a merge and generate indexed format profile
4) use that profile with the trunk compiler

However, the other direction does not work -- you can not expect older
version of compiler to consume indexed format profile of the new
compiler.


> llvm-profdata cannot read raw profile data files generated by compiler-rt
> 3.7. Is this intended?

yes. New version of raw profile reader does not understand old version
of raw profile data. You need to use older version of llvm-profdata to
produce old version of indexed format, and then use new version of
llvm-profdata to convert the indexed format into new version. Here is
flow:

1) llvm-profdata_3.7 merge -o my_profdata_old.profdata my_profdata_old.profraw

At this point, my_profdata_old.profdata *can* be consumed by trunk
version of compiler already, but if you want to convert it to new
version of indexed format, do this:

2) llvm-profdata_trunk merge -o my_profdata_new.profdata
my_profdata_old.profdata

Now my_profdata_new.profdata is in shiny new version (indexed).


 Is llvm-profdata's output also incompatible between
> different versions of LLVM (InstrProfWriter/Reader are not meant to be
> compatible across LLVM versions) ?

The merge output format (indexed format) is backward compatible as
explained by Vedant.

>
> LDC is currently compatible with LLVM 3.5 up to trunk. To support PGO for
> multiple LLVM versions (>= 3.7), I think that we will need to have a copy of
> compiler-rt/lib/profile for each version of LLVM >= 3.7. We also need to
> have a copy of llvm-profdata for each version of LLVM >= 3.7. We already do
> this with the FileCheck utility and will not be a big problem.
>

yes -- see above. Remember compiler instrumentation and supporting
runtime must be in sync.

thanks,

David


> Does that make sense? What do you advise?
>
> Thanks very much for your comments,
>   Johan
>