[PATCH] D20715: [docs] Document the source-based code coverage feature
Vedant Kumar via cfe-commits
cfe-commits at lists.llvm.org
Wed Jun 1 17:58:41 PDT 2016
> On Jun 1, 2016, at 11:30 AM, Justin Bogner <mail at justinbogner.com> wrote:
>
> Vedant Kumar <vsk at apple.com> writes:
>> vsk created this revision.
>> vsk added a reviewer: bogner.
>> vsk added subscribers: kcc, cfe-commits, silvas.
>>
>> It would be helpful to have a user-friendly guide for code
>> coverage. There is some overlap with [1], but this document visits
>> issues which may affect users in more depth.
>>
>> Prompted by: https://llvm.org/bugs/show_bug.cgi?id=27781
>>
>> [1] http://llvm.org/docs/CoverageMappingFormat.html
> ...
>> vsk updated this revision to Diff 59078.
>> vsk marked an inline comment as done.
>> vsk added a comment.
>>
>> - Actually link in the new document into Index.rst.
>
> A couple of comments below, but since this is prose it's basically all
> just opinion. Feel free to commit any time - we can always make
> improvements in tree later.
Thanks for all the feedback! I've incorporated all of it with minor
changes.
Let's use r271454 as a starting point, then.
vedant
>
>>
>> http://reviews.llvm.org/D20715
>>
>> Files:
>> docs/SourceBasedCodeCoverage.rst
>> docs/index.rst
>>
>> Index: docs/index.rst
>> ===================================================================
>> --- docs/index.rst
>> +++ docs/index.rst
>> @@ -33,6 +33,7 @@
>> ControlFlowIntegrity
>> LTOVisibility
>> SafeStack
>> + SourceBasedCodeCoverage
>> Modules
>> MSVCCompatibility
>> CommandGuide/index
>> Index: docs/SourceBasedCodeCoverage.rst
>> ===================================================================
>> --- /dev/null
>> +++ docs/SourceBasedCodeCoverage.rst
>> @@ -0,0 +1,187 @@
>> +==========================
>> +Source-based Code Coverage
>> +==========================
>> +
>> +.. contents::
>> + :local:
>> +
>> +Introduction
>> +============
>> +
>> +This document explains how to use clang's source-based code coverage feature.
>> +It's called "source-based" because it operates on AST and preprocessor
>> +information directly. This allows it to generate very precise coverage data.
>> +
>> +Clang ships two other code coverage implementations:
>> +
>> +* :doc:`SanitizerCoverage` - A low-overhead tool meant for use alongside the
>> + various sanitizers. It can provide up to edge-level coverage.
>> +
>> +* gcov - A GCC-compatible coverage implementation which operates on DebugInfo.
>> +
>> +From this point onwards "code coverage" will refer to the source-based kind.
>> +
>> +The code coverage workflow
>> +==========================
>> +
>> +The code coverage workflow consists of three main steps:
>> +
>> +1. Compiling with coverage enabled.
>> +
>> +2. Running the instrumented program.
>> +
>> +3. Creating coverage reports.
>> +
>> +The next few sections work through a complete, copy-'n-paste friendly example
>> +based on this program:
>> +
>> +.. code-block:: console
>> +
>> + % cat <<EOF > foo.cc
>> + #define BAR(x) ((x) || (x))
>> + template <typename T> void foo(T x) {
>> + for (unsigned I = 0; I < 10; ++I) { BAR(I); }
>> + }
>> + int main() {
>> + foo<int>(0);
>> + foo<float>(0);
>> + return 0;
>> + }
>> + EOF
>> +
>> +Compiling with coverage enabled
>> +===============================
>> +
>> +To compile code with coverage enabled pass ``-fprofile-instr-generate
>> +-fcoverage-mapping`` to the compiler:
>> +
>> +.. code-block:: console
>> +
>> + # Step 1: Compile with coverage enabled.
>> + % clang++ -fprofile-instr-generate -fcoverage-mapping foo.cc -o foo
>> +
>> +Note that linking together code with and without coverage instrumentation is
>> +supported: any uninstrumented code simply won't be accounted for.
>> +
>> +Running the instrumented program
>> +================================
>> +
>> +The next step is to run the instrumented program. When the program exits it
>> +will write a **raw profile** to the path specified by the ``LLVM_PROFILE_FILE``
>> +environment variable. If that variable does not exist the profile is written to
>> +``./default.profraw``.
>
> Something like "``default.profraw`` in the current directory of the
> program" might be clearer.
>
>> +
>> +If ``LLVM_PROFILE_FILE`` contains a path to a non-existent directory the
>> +missing directory structure will be created. Additionally, the following
>> +special **pattern strings** are replaced:
>> +
>> +* "%p" expands out to the PID.
>
> Maybe spell out process ID?
>
>> +
>> +* "%h" expands out to the hostname of the machine running the program.
>> +
>> +.. code-block:: console
>> +
>> + # Step 2: Run the program.
>> + % LLVM_PROFILE_FILE="foo.profraw" ./foo
>> +
>> +Creating coverage reports
>> +=========================
>> +
>> +Raw profiles have to be **indexed** before they can be used to generated
>> +coverage reports:
>> +
>> +.. code-block:: console
>> +
>> + # Step 3(a): Index the raw profile.
>> + % llvm-profdata merge -sparse foo.profraw -o foo.profdata
>> +
>> +You may be wondering why raw profiles aren't indexed automatically. In
>> +real-world projects multiple profiles have to be merged together before a
>> +report can be generated. This merge step is inevitable, so it makes sense to
>> +handle the compute-intensive indexing process at that point. A separate
>> +indexing step has the added benefit of keeping the compiler runtime small and
>> +simple.
>
> This comment feels like it comes out of nowhere, and I'm not really
> convinced it adds much. If you feel strongly that we need to ward off
> "why do I need to do this if I only have one raw profile" questions, I'd
> probably phrase it something like this:
>
> This merge step is necessary even if you only have one raw profile file.
> Separating the indexing step like this allows the compiler runtime to be
> efficient and simple, whereas most users of coverage will need to merge
> multiple profiles anyway.
>
>> +
>> +There are multiple different ways to render coverage reports. One option is to
>> +generate a line-oriented report:
>> +
>> +.. code-block:: console
>> +
>> + # Step 3(b): Create a line-oriented coverage report.
>> + % llvm-cov show ./foo -instr-profile=foo.profdata
>> +
>> +This report includes a summary view as well as dedicated sub-views for
>> +templated functions and their instantiations. For our example program, we get
>> +distinct views for ``foo<int>(...)`` and ``foo<float>(...)``. If
>> +``-show-line-counts-or-regions`` is enabled, ``llvm-cov`` displays sub-line
>> +region counts (even in macro expansions):
>> +
>> +.. code-block:: console
>> +
>> + 20| 1|#define BAR(x) ((x) || (x))
>> + ^20 ^2
>> + 2| 2|template <typename T> void foo(T x) {
>> + 22| 3| for (unsigned I = 0; I < 10; ++I) { BAR(I); }
>> + ^22 ^20 ^20^20
>> + 2| 4|}
>> + ------------------
>> + | _Z3fooIiEvT_:
>
> People always ask about the C++ mangled names. Maybe we should mention
> the option of running the results through c++filt? You'd probably also
> want to mention the -use-color flag, since piping the output to c++filt
> will disable the automatic detection.
>
> Personally, I find the following command useful enough to mention
> briefly:
>
> llvm-cov show -use-color <...> | c++filt | less -R
>
>> + | 1| 2|template <typename T> void foo(T x) {
>> + | 11| 3| for (unsigned I = 0; I < 10; ++I) { BAR(I); }
>> + | ^11 ^10 ^10^10
>> + | 1| 4|}
>> + ------------------
>> + | _Z3fooIfEvT_:
>> + | 1| 2|template <typename T> void foo(T x) {
>> + | 11| 3| for (unsigned I = 0; I < 10; ++I) { BAR(I); }
>> + | ^11 ^10 ^10^10
>> + | 1| 4|}
>> + ------------------
>> +
>> +It's possible to generate a file-level summary of coverage statistics (instead
>> +of a line-oriented report) with:
>> +
>> +.. code-block:: console
>> +
>> + # Step 3(c): Create a coverage summary.
>> + % llvm-cov report ./foo -instr-profile=foo.profdata
>> + Filename Regions Miss Cover Functions Executed
>> + -----------------------------------------------------------------------
>> + /tmp/foo.cc 13 0 100.00% 3 100.00%
>> + -----------------------------------------------------------------------
>> + TOTAL 13 0 100.00% 3 100.00%
>> +
>> +A few final notes:
>> +
>> +* The ``-sparse`` flag is optional but can result in dramatically smaller
>> + indexed profiles. This option should not be used if the indexed profile will
>> + be reused for PGO.
>> +
>> +* Raw profiles can be discarded after they are indexed. Advanced use of the
>> + profile runtime library allows an instrumented program to merge profiling
>> + information directly into an existing raw profile on disk. The details are
>> + out of scope.
>> +
>> +* The ``llvm-profdata`` tool can be used to merge together multiple raw or
>> + indexed profiles. To combine profiling data from multiple runs of a program,
>> + try e.g:
>> +
>> +.. code-block:: console
>> +
>> + % llvm-profdata merge -sparse foo1.profraw foo2.profdata -o foo3.profdata
>> +
>> +Format compatibility guarantees
>> +===============================
>> +
>> +* There are no backwards or forwards compatibility guarantees for the raw
>> + profile format. Raw profiles may be dependent on the specific compiler
>> + revision used to generate them. It's inadvisable to store raw profiles for
>> + long periods of time.
>> +
>> +* Tools must retain **backwards** compatibility with indexed profile formats.
>> + These formats are not forwards-compatible: i.e, a tool which uses format
>> + version X will not be able to understand format version (X+k).
>> +
>> +* There is a third format in play: the format of the coverage mappings emitted
>> + into instrumented binaries. Tools must retain **backwards** compatibility
>> + with these formats. These formats are not forwards-compatible.
>
More information about the cfe-commits
mailing list