[PATCH] D20993: [lit] Add support for PGO profile and code coverage collection

[Xinliang David Li via llvm-commits]

On Fri, Jun 3, 2016 at 9:32 PM, Matthias Braun <matze at braunis.de> wrote:

> MatzeB added a comment.
>
> In http://reviews.llvm.org/D20993#449116, @vsk wrote:
>
> > In http://reviews.llvm.org/D20993#449092, @MatzeB wrote:
> >
> > > This adds a bigger bunch of code to lit, I wonder if it is necessary:
> > >
> > > - I assume the necessity for the logic inside lit is just the fact
> that the profile data overwrites each other after running a command. Is
> setting LLVM_PROFILE_FILE with a '%p' placeholder not enough to avoid this?
> >
> >
> > Using PID substitution gets close to solving the problem of having
> overwritten profiles, but not all the way. On 32-bit systems PID wraparound
> would pose a real problem. In this patch I include the hash of the test
> command to minimize loss of profiles.
>
>
> This is a really annoying problem to have... Maybe we should find a
> solution within the profile infrastructure itself (can we add another flag
> that adds a unique suffix to the filename if it already exists?) so not
> every user of the profiling infrastructure has to jump through the same
> hoops (I've done a similar dance in the test-suite profile support).
>
> >
>
> >
>
> > > The final profile data merging can always be done outside of llvm-lit
> afterwards.
>
> >
>
> >
>
> > On a practical level, I don't think this is possible. Raw profiles are
> too large. Turning off the cleanup step and running check-llvm produces
> over half a terrabyte of data. There are a few ways to address this without
> touching lit:
>
> >
>
> > 1. Reduce the size of raw profiles. This would make the compiler runtime
> larger and more complex.
>
> > 2. Run a monitor process that does the merging/cleanup. I don't think
> that approach has any advantages compared to modifying lit.
>
> >
>
> >   Alternatively, we could do in-place raw profile merging in the
> compiler runtime. I don't think that's preferable because we'd have to
> introduce a lot of complexity to compiler-rt (e.g portable mandatory file
> locking).
>
>
> Ah, so that seems harder to solve without an external driver merging the
> profile data. I'm still not a big fan of injecting this stuff into the core
> of lit, but we may have no other choice today.
>

This is handled by the compiler-rt -- it should be totally transparent to
the user.

David




>
> >
>
> >
>
> > > Or are there other reasons why you need to intercept the opt/llc/etc.
> calls?
>
> >
>
> >
>
> > Intercepting the calls makes it possible to generate 'unique' hashes for
> the profiles, see my comment above.
>
> >
>
> > > - The lit tests don't strike me as the most typical llvm usage (they
> are often designed to test corner cases of the compiler), wouldn't it be
> better to gather profiles by compiling the llvm test-suite or something
> similar?
>
> >
>
> >
>
> > It's true that the profiles gathered from check-llvm aren't ideal for
> PGO. But, they are perfect for measuring code coverage of the llvm
> codebase. Moreover because lit is portable, other projects can use these
> changes to gather coverage reports with their tests.
>
>
> Oh indeed I did not think about code coverage.
>
> > (IIRC the test-suite also uses lit?)
>
>
> The test-suite uses lit to execute the benchmarks. I already added
> profiling support for that (see the
> TEST_SUITE_PROFILE_GENERATE/TEST_SUITE_PROFILE_USE variants). This is
> however about running the benchmarks. It does not help you when you want to
> generate profile data from the clang executable that compiled the
> benchmarks as that is only done by make (or ninja) and not lit.
>
>
> http://reviews.llvm.org/D20993
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20160603/79299b25/attachment.html>