[llvm-dev] [EXTERNAL] How to get branch coverage by using 'source-based code coverage'

Wed May 6 09:03:43 PDT 2020

Hi Ted,

I’ll attempt to answer your questions:

1.)    What's the "official" evaluation formula on branch coverage? Sorry I can't find any compelling documents to describe the accurate algorithm on this issue. I just thought gcov/lcov won't consider how many times the decision evaluated to False.

a.      [Alan]: I share your concern here. My reference is the ISO26262 standard for functional safety, which defines branch coverage as “percentage of branches of the control flow of a computer program executed during a test.  Note 1: 100% branch coverage implies 100% statement coverage., Note 2: An if-statement always has two branches – condition true and condition false – independent of the existence of an else-clause”

                                                               i.      Basically, condition true and condition false for every decision in a program.  In contrast, MC/DC applies to single condition outcomes that affect a decision.

b.      [Alan] Your GCOV output is producing branch information for each condition of the decision, and each of those branches corresponds to the true and false evaluation of each condition. The problem with GCC/GCOV is that, because of the way it instruments, it’s not obvious how branches correspond to the source code.

2.)    Are there any existing post-processing tools based on clang/llvm source based coverage

a.       [Alan] I’m not aware of them, but others may know

3.)    How to disable STL (or other) noise in gcov branch coverage?

a.       [Alan] I think this is a perennial problem – gcov reports hidden branches that are created, but the data isn’t necessarily useful from a source-code coverage perspective.

-Alan

From: LoveZhi Xie [mailto:crazyforzhi1987 at gmail.com]
Sent: Monday, May 4, 2020 3:24 AM
To: Phipps, Alan
Cc: llvm-dev at lists.llvm.org; Vedant Kumar
Subject: Re: [EXTERNAL] [llvm-dev] How to get branch coverage by using 'source-based code coverage'

Hi, Alan

Thanks for making it clear. But I was more confused now :(

I tested on a simple program and used both gcov and lcov to get branch coverage.
The code and build commands as below:

Example simple.cc
#include <string>
// If not comment this line, the branch coverage won't reach to 100%
// #include <iostream>
int main(int argc, const char* argv[]) {
  std::string str = "foo";
  str = argv[1];
  if (str == "foo" || str == "bar") {
    int a = 5;
    a = 7;
  } else {
    int b = 6;
    b = 4;
  }
  return 0;
}

Coverage build commands
#!/bin/bash
g++ -o simple -fno-exceptions -fno-inline -fprofile-arcs -ftest-coverage simple.cc
./simple "foo"
./simple "bar"
./simple "hello1"
./simple "hello2"
./simple "hello3"
./simple "hello4"
./simple "hello5"
./simple "hello6"
./simple "hello7"
./simple "hello8"
lcov --rc lcov_branch_coverage=1 --capture --directory . --output-file simple.lcov.info<http://simple.lcov.info>
genhtml simple.lcov.info<http://simple.lcov.info> --function-coverage --branch-coverage --output-directory lcov_out/
gcov -b -c simple.gcno

Both gcov and lcov will give a 100% branch coverage report:
[cov1.png]　[cov2.png]

My questions:

  *    What's the "official" evaluation formula on branch coverage?
            Sorry I can't find any compelling documents to describe the accurate algorithm on this issue.
            I just thought gcov/lcov won't consider how many times the decision evaluated to False.
          Please see the gcov file on my example: simple.cc. There are 6 branches in total which are all hit
            So the result would be 6/6 * 100% = 100%

            [cov3.png]

  *   Are there any existing post-processing tools based on clang/llvm source based coverage?
           I just saw a clion plugin<https://github.com/zero9178/C-Cpp-Coverage-for-CLion> at github which may generate branch coverage by using region coverage data. But this depends on clion
           which is not easy to implement automation test.

  *   How to disable STL (or other) noise in gcov branch coverage?
           Sorry, this is not clang/llvm question. But I am still looking forward to some suggestions. For example, if I add iostream header back to
           simple.cc, then we won't get a 100% branch coverage. This may because iostream will introduce some global destruction branch.

           [cov4.png]

           Or if it's impossible, can we think clang region coverage should be a best practice in C++ testing world??

　　　AFAIK, chromium is using region coverage now.
　　　[cov5.png]

FYI.

Thanks,
Ted Xie

Phipps, Alan <a-phipps at ti.com<mailto:a-phipps at ti.com>> 于2020年5月4日周一 上午12:54写道：
+ Vedant Kumar

Hi Ted!

I would not say that clang region coverage is branch coverage(‘aka decision coverage’).  I would say that it may be possible to extrapolate branch decision coverage based on region coverage.  For example, consider the following:

[region0]
if (decision) {
   [region1]
}

Region coverage will tell you that whether [region1] was executed, and this will be enough to tell you that the branch decision evaluated to True at least once.  However, this will not tell you how many times the decision evaluated to False, which is also important when considering branch coverage.  In this case, you can calculate this value by subtracting the number of times [region0] was executed from the number of times [region1] was executed.

What I am implementing for branch condition coverage will do this for you but will provide more granular True/False counts for individual conditions that comprise all Boolean expression (which also generate branches), and will tally all of these cases as branches that are either covered or not covered.

I hope that helps.

As for the future goals, I do plan on addressing what’s required for MC/DC, but that is work that I have not yet planned.

Thanks!

-Alan Phipps

From: LoveZhi Xie [mailto:crazyforzhi1987 at gmail.com<mailto:crazyforzhi1987 at gmail.com>]
Sent: Sunday, May 3, 2020 11:12 AM
To: Phipps, Alan
Cc: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
Subject: Re: [EXTERNAL] [llvm-dev] How to get branch coverage by using 'source-based code coverage'

Hi, Alan

Really very excited to receive your email and sorry to be slow replying, it has been exceptionally busy over the last few days ;(

Your explanation made the problem clear to me. So gcov branch coverage should be called condition coverage and clang region coverage
is branch coverage in fact(also known as decision/C1), right?

And llvm/clang will support all the following coverage criteria in future

  *   Line coverage
  *   Function coverage
  *   Branch coverage/region coverage
  *   Condition coverage
  *   Modified Condition/Decision Coverage
If there is any misunderstanding here, please point it out.

Thanks,
Ted Xie

Phipps, Alan <a-phipps at ti.com<mailto:a-phipps at ti.com>> 于2020年4月28日周二 下午10:14写道：
Hi Ted,

Presently, branch coverage is not supported in clang source-based code coverage, but I am working on implementing branch condition coverage right now.  It will track True/False branches for each leaf-level condition, including within Boolean expressions comprised of logical operators (“&&”, “||”).  Technically, this condition-based coverage is more granular than how “branch coverage” is often defined (tracking on that control flow decisions allow for all regions to be hit), and is closer to the granularity that GCOV provides, but it’s anchored to the source code, won’t count “hidden branches”, and isn’t negatively impacted by optimization (although the reverse isn’t true: coverage instrumentation may impact optimization).

My work is a step closer to getting us toward Modified Condition/Decision Coverage (MC/DC).  I hope to upstream my work within the next few months, so unfortunately it won’t help you in the short-term.  Region coverage may be enough for what you need if you can ascertain based on the coverage that control-flow decisions have evaluated to both true and false, but I’ll let others comment here with suggestions.

-Alan Phipps

From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org<mailto:llvm-dev-bounces at lists.llvm.org>] On Behalf Of LoveZhi Xie via llvm-dev
Sent: Sunday, April 26, 2020 11:36 AM
To: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
Subject: [EXTERNAL] [llvm-dev] How to get branch coverage by using 'source-based code coverage'

Hi, llvm/clang experts

I need to get the branch coverage for some testing code. But i found gcov can't give a expected coverage which may
count some 'hidden branch' in (See stackoverflow answer<https://stackoverflow.com/questions/42003783/lcov-gcov-branch-coverage-with-c-producing-branches-all-over-the-place>). Instead, I turn to use clang and the 'source-based code coverage' feature
may be a right choice. But i can't find anything to describe branch coverage explicitly on the official site<https://clang.llvm.org/docs/SourceBasedCodeCoverage.html>.

So how to use clang to get branch coverage by utilizing 'source-based...' feature?  And is region coverage equivalent to branch coverage?
(I tested on a simple program and region coverage seems pretty close to branch coverage)

Any help is highly appreciated

Thanks,
Ted Xie

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200506/5945e5b2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 33987 bytes
Desc: image002.png
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200506/5945e5b2/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 18185 bytes
Desc: image004.png
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200506/5945e5b2/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.jpg
Type: image/jpeg
Size: 10451 bytes
Desc: image006.jpg
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200506/5945e5b2/attachment-0003.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image008.jpg
Type: image/jpeg
Size: 6144 bytes
Desc: image008.jpg
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200506/5945e5b2/attachment-0004.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image010.jpg
Type: image/jpeg
Size: 13518 bytes
Desc: image010.jpg
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200506/5945e5b2/attachment-0005.jpg>