[PATCH] D20715: [docs] Document the source-based code coverage feature

Justin Bogner via cfe-commits cfe-commits at lists.llvm.org
Wed Jun 1 11:30:55 PDT 2016


Vedant Kumar <vsk at apple.com> writes:
> vsk created this revision.
> vsk added a reviewer: bogner.
> vsk added subscribers: kcc, cfe-commits, silvas.
>
> It would be helpful to have a user-friendly guide for code
> coverage. There is some overlap with [1], but this document visits
> issues which may affect users in more depth.
>
> Prompted by: https://llvm.org/bugs/show_bug.cgi?id=27781
>
> [1] http://llvm.org/docs/CoverageMappingFormat.html
 ...
> vsk updated this revision to Diff 59078.
> vsk marked an inline comment as done.
> vsk added a comment.
>
> - Actually link in the new document into Index.rst.

A couple of comments below, but since this is prose it's basically all
just opinion. Feel free to commit any time - we can always make
improvements in tree later.

>
> http://reviews.llvm.org/D20715
>
> Files:
>   docs/SourceBasedCodeCoverage.rst
>   docs/index.rst
>
> Index: docs/index.rst
> ===================================================================
> --- docs/index.rst
> +++ docs/index.rst
> @@ -33,6 +33,7 @@
>     ControlFlowIntegrity
>     LTOVisibility
>     SafeStack
> +   SourceBasedCodeCoverage
>     Modules
>     MSVCCompatibility
>     CommandGuide/index
> Index: docs/SourceBasedCodeCoverage.rst
> ===================================================================
> --- /dev/null
> +++ docs/SourceBasedCodeCoverage.rst
> @@ -0,0 +1,187 @@
> +==========================
> +Source-based Code Coverage
> +==========================
> +
> +.. contents::
> +   :local:
> +
> +Introduction
> +============
> +
> +This document explains how to use clang's source-based code coverage feature.
> +It's called "source-based" because it operates on AST and preprocessor
> +information directly. This allows it to generate very precise coverage data.
> +
> +Clang ships two other code coverage implementations:
> +
> +* :doc:`SanitizerCoverage` - A low-overhead tool meant for use alongside the
> +  various sanitizers. It can provide up to edge-level coverage.
> +
> +* gcov - A GCC-compatible coverage implementation which operates on DebugInfo.
> +
> +From this point onwards "code coverage" will refer to the source-based kind.
> +
> +The code coverage workflow
> +==========================
> +
> +The code coverage workflow consists of three main steps:
> +
> +1. Compiling with coverage enabled.
> +
> +2. Running the instrumented program.
> +
> +3. Creating coverage reports.
> +
> +The next few sections work through a complete, copy-'n-paste friendly example
> +based on this program:
> +
> +.. code-block:: console
> +
> +    % cat <<EOF > foo.cc
> +    #define BAR(x) ((x) || (x))
> +    template <typename T> void foo(T x) {
> +      for (unsigned I = 0; I < 10; ++I) { BAR(I); }
> +    }
> +    int main() {
> +      foo<int>(0);
> +      foo<float>(0);
> +      return 0;
> +    }
> +    EOF
> +
> +Compiling with coverage enabled
> +===============================
> +
> +To compile code with coverage enabled pass ``-fprofile-instr-generate
> +-fcoverage-mapping`` to the compiler:
> +
> +.. code-block:: console
> +
> +    # Step 1: Compile with coverage enabled.
> +    % clang++ -fprofile-instr-generate -fcoverage-mapping foo.cc -o foo
> +
> +Note that linking together code with and without coverage instrumentation is
> +supported: any uninstrumented code simply won't be accounted for.
> +
> +Running the instrumented program
> +================================
> +
> +The next step is to run the instrumented program. When the program exits it
> +will write a **raw profile** to the path specified by the ``LLVM_PROFILE_FILE``
> +environment variable. If that variable does not exist the profile is written to
> +``./default.profraw``.

Something like "``default.profraw`` in the current directory of the
program" might be clearer.

> +
> +If ``LLVM_PROFILE_FILE`` contains a path to a non-existent directory the
> +missing directory structure will be created.  Additionally, the following
> +special **pattern strings** are replaced:
> +
> +* "%p" expands out to the PID.

Maybe spell out process ID?

> +
> +* "%h" expands out to the hostname of the machine running the program.
> +
> +.. code-block:: console
> +
> +    # Step 2: Run the program.
> +    % LLVM_PROFILE_FILE="foo.profraw" ./foo
> +
> +Creating coverage reports
> +=========================
> +
> +Raw profiles have to be **indexed** before they can be used to generated
> +coverage reports:
> +
> +.. code-block:: console
> +
> +    # Step 3(a): Index the raw profile.
> +    % llvm-profdata merge -sparse foo.profraw -o foo.profdata
> +
> +You may be wondering why raw profiles aren't indexed automatically. In
> +real-world projects multiple profiles have to be merged together before a
> +report can be generated. This merge step is inevitable, so it makes sense to
> +handle the compute-intensive indexing process at that point. A separate
> +indexing step has the added benefit of keeping the compiler runtime small and
> +simple.

This comment feels like it comes out of nowhere, and I'm not really
convinced it adds much. If you feel strongly that we need to ward off
"why do I need to do this if I only have one raw profile" questions, I'd
probably phrase it something like this:

  This merge step is necessary even if you only have one raw profile file.
  Separating the indexing step like this allows the compiler runtime to be
  efficient and simple, whereas most users of coverage will need to merge
  multiple profiles anyway.

> +
> +There are multiple different ways to render coverage reports. One option is to
> +generate a line-oriented report:
> +
> +.. code-block:: console
> +
> +    # Step 3(b): Create a line-oriented coverage report.
> +    % llvm-cov show ./foo -instr-profile=foo.profdata
> +
> +This report includes a summary view as well as dedicated sub-views for
> +templated functions and their instantiations. For our example program, we get
> +distinct views for ``foo<int>(...)`` and ``foo<float>(...)``.  If
> +``-show-line-counts-or-regions`` is enabled, ``llvm-cov`` displays sub-line
> +region counts (even in macro expansions):
> +
> +.. code-block:: console
> +
> +       20|    1|#define BAR(x) ((x) || (x))
> +                               ^20     ^2
> +        2|    2|template <typename T> void foo(T x) {
> +       22|    3|  for (unsigned I = 0; I < 10; ++I) { BAR(I); }
> +                                       ^22     ^20  ^20^20
> +        2|    4|}
> +    ------------------
> +    | _Z3fooIiEvT_:

People always ask about the C++ mangled names. Maybe we should mention
the option of running the results through c++filt? You'd probably also
want to mention the -use-color flag, since piping the output to c++filt
will disable the automatic detection. 

Personally, I find the following command useful enough to mention
briefly:

  llvm-cov show -use-color <...> | c++filt | less -R

> +    |      1|    2|template <typename T> void foo(T x) {
> +    |     11|    3|  for (unsigned I = 0; I < 10; ++I) { BAR(I); }
> +    |                                     ^11     ^10  ^10^10
> +    |      1|    4|}
> +    ------------------
> +    | _Z3fooIfEvT_:
> +    |      1|    2|template <typename T> void foo(T x) {
> +    |     11|    3|  for (unsigned I = 0; I < 10; ++I) { BAR(I); }
> +    |                                     ^11     ^10  ^10^10
> +    |      1|    4|}
> +    ------------------
> +
> +It's possible to generate a file-level summary of coverage statistics (instead
> +of a line-oriented report) with:
> +
> +.. code-block:: console
> +
> +    # Step 3(c): Create a coverage summary.
> +    % llvm-cov report ./foo -instr-profile=foo.profdata
> +    Filename                    Regions    Miss   Cover Functions  Executed
> +    -----------------------------------------------------------------------
> +    /tmp/foo.cc                      13       0 100.00%         3   100.00%
> +    -----------------------------------------------------------------------
> +    TOTAL                            13       0 100.00%         3   100.00%
> +
> +A few final notes:
> +
> +* The ``-sparse`` flag is optional but can result in dramatically smaller
> +  indexed profiles. This option should not be used if the indexed profile will
> +  be reused for PGO.
> +
> +* Raw profiles can be discarded after they are indexed. Advanced use of the
> +  profile runtime library allows an instrumented program to merge profiling
> +  information directly into an existing raw profile on disk. The details are
> +  out of scope.
> +
> +* The ``llvm-profdata`` tool can be used to merge together multiple raw or
> +  indexed profiles. To combine profiling data from multiple runs of a program,
> +  try e.g:
> +
> +.. code-block:: console
> +
> +    % llvm-profdata merge -sparse foo1.profraw foo2.profdata -o foo3.profdata
> +
> +Format compatibility guarantees
> +===============================
> +
> +* There are no backwards or forwards compatibility guarantees for the raw
> +  profile format. Raw profiles may be dependent on the specific compiler
> +  revision used to generate them. It's inadvisable to store raw profiles for
> +  long periods of time.
> +
> +* Tools must retain **backwards** compatibility with indexed profile formats.
> +  These formats are not forwards-compatible: i.e, a tool which uses format
> +  version X will not be able to understand format version (X+k).
> +
> +* There is a third format in play: the format of the coverage mappings emitted
> +  into instrumented binaries. Tools must retain **backwards** compatibility
> +  with these formats. These formats are not forwards-compatible.



More information about the cfe-commits mailing list