[llvm-dev] [EXTERNAL] How to get branch coverage by using 'source-based code coverage'

[Phipps, Alan via llvm-dev]

+ Vedant Kumar

Hi Ted!

I would not say that clang region coverage is branch coverage(‘aka decision coverage’).  I would say that it may be possible to extrapolate branch decision coverage based on region coverage.  For example, consider the following:

[region0]
if (decision) {
   [region1]
}

Region coverage will tell you that whether [region1] was executed, and this will be enough to tell you that the branch decision evaluated to True at least once.  However, this will not tell you how many times the decision evaluated to False, which is also important when considering branch coverage.  In this case, you can calculate this value by subtracting the number of times [region0] was executed from the number of times [region1] was executed.

What I am implementing for branch condition coverage will do this for you but will provide more granular True/False counts for individual conditions that comprise all Boolean expression (which also generate branches), and will tally all of these cases as branches that are either covered or not covered.

I hope that helps.

As for the future goals, I do plan on addressing what’s required for MC/DC, but that is work that I have not yet planned.

Thanks!

-Alan Phipps

From: LoveZhi Xie [mailto:crazyforzhi1987 at gmail.com]
Sent: Sunday, May 3, 2020 11:12 AM
To: Phipps, Alan
Cc: llvm-dev at lists.llvm.org
Subject: Re: [EXTERNAL] [llvm-dev] How to get branch coverage by using 'source-based code coverage'

Hi, Alan

Really very excited to receive your email and sorry to be slow replying, it has been exceptionally busy over the last few days ;(

Your explanation made the problem clear to me. So gcov branch coverage should be called condition coverage and clang region coverage
is branch coverage in fact(also known as decision/C1), right?

And llvm/clang will support all the following coverage criteria in future

  *   Line coverage
  *   Function coverage
  *   Branch coverage/region coverage
  *   Condition coverage
  *   Modified Condition/Decision Coverage
If there is any misunderstanding here, please point it out.

Thanks,
Ted Xie



Phipps, Alan <a-phipps at ti.com<mailto:a-phipps at ti.com>> 于2020年4月28日周二 下午10:14写道：
Hi Ted,

Presently, branch coverage is not supported in clang source-based code coverage, but I am working on implementing branch condition coverage right now.  It will track True/False branches for each leaf-level condition, including within Boolean expressions comprised of logical operators (“&&”, “||”).  Technically, this condition-based coverage is more granular than how “branch coverage” is often defined (tracking on that control flow decisions allow for all regions to be hit), and is closer to the granularity that GCOV provides, but it’s anchored to the source code, won’t count “hidden branches”, and isn’t negatively impacted by optimization (although the reverse isn’t true: coverage instrumentation may impact optimization).

My work is a step closer to getting us toward Modified Condition/Decision Coverage (MC/DC).  I hope to upstream my work within the next few months, so unfortunately it won’t help you in the short-term.  Region coverage may be enough for what you need if you can ascertain based on the coverage that control-flow decisions have evaluated to both true and false, but I’ll let others comment here with suggestions.

-Alan Phipps

From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org<mailto:llvm-dev-bounces at lists.llvm.org>] On Behalf Of LoveZhi Xie via llvm-dev
Sent: Sunday, April 26, 2020 11:36 AM
To: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
Subject: [EXTERNAL] [llvm-dev] How to get branch coverage by using 'source-based code coverage'

Hi, llvm/clang experts

I need to get the branch coverage for some testing code. But i found gcov can't give a expected coverage which may
count some 'hidden branch' in (See stackoverflow answer<https://stackoverflow.com/questions/42003783/lcov-gcov-branch-coverage-with-c-producing-branches-all-over-the-place>). Instead, I turn to use clang and the 'source-based code coverage' feature
may be a right choice. But i can't find anything to describe branch coverage explicitly on the official site<https://clang.llvm.org/docs/SourceBasedCodeCoverage.html>.

So how to use clang to get branch coverage by utilizing 'source-based...' feature?  And is region coverage equivalent to branch coverage?
(I tested on a simple program and region coverage seems pretty close to branch coverage)

Any help is highly appreciated

Thanks,
Ted Xie


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200503/063e5bdc/attachment-0001.html>