Hi, Alan
Thanks for making it clear. But I was more confused now :(
I tested on a simple program and used both gcov and lcov to get branch
coverage.
The code and build commands as below:
*Example simple.cc*
#include <string>
// If not comment this line, the branch coverage won't reach to 100%
// #include <iostream>
int main(int argc, const char* argv[]) {
std::string str = "foo";
str = argv[1];
if (str == "foo" || str == "bar") {
int a = 5;
a = 7;
} else {
int b = 6;
b = 4;
}
return 0;
}
*Coverage build commands*
#!/bin/bash
g++ -o simple -fno-exceptions -fno-inline -fprofile-arcs -ftest-coverage
simple.cc
./simple "foo"
./simple "bar"
./simple "hello1"
./simple "hello2"
./simple "hello3"
./simple "hello4"
./simple "hello5"
./simple "hello6"
./simple "hello7"
./simple "hello8"
lcov --rc lcov_branch_coverage=1 --capture --directory . --output-file
simple.lcov.info
genhtml simple.lcov.info --function-coverage --branch-coverage
--output-directory lcov_out/
gcov -b -c simple.gcno
Both gcov and lcov will give a 100% branch coverage report:
[image: cov1.png] [image: cov2.png]
My questions:
- What's the "*official*" evaluation formula on branch coverage?
Sorry I can't find any compelling documents to describe the
accurate algorithm on this issue.
I just thought gcov/lcov won't consider *how many times the
decision evaluated to False.*
* P*lease see the gcov file on my example: simple.cc. There are 6
branches in total which are all hit
So the result would be 6/6 * 100% = 100%
[image: cov3.png]
- Are there any existing post-processing tools based on clang/llvm
source based coverage?
I just saw a clion plugin
<https://github.com/zero9178/C-Cpp-Coverage-for-CLion> at github which may
generate branch coverage by using region coverage data. But this depends on
clion
which is not easy to implement automation test.
- How to disable STL (or other) noise in gcov branch coverage?
Sorry, this is not clang/llvm question. But I am still looking
forward to some suggestions. For example, if I add iostream header back to
simple.cc, then we won't get a 100% branch coverage. This may
because iostream will introduce some global destruction branch.
[image: cov4.png]
Or if it's impossible, can we think clang region coverage should
be a best practice *in C++ testing world*??
AFAIK, chromium is using region coverage now.
[image: cov5.png]
FYI.
Thanks,
Ted Xie
Phipps, Alan <a-phipps at ti.com> 于2020年5月4日周一 上午12:54写道：
> + Vedant Kumar
>
>
>
> Hi Ted!
>
>
>
> I would not say that clang region coverage is branch coverage(‘aka
> decision coverage’). I would say that it may be possible to extrapolate
> branch decision coverage based on region coverage. For example, consider
> the following:
>
>
>
> [region0]
>
> if (decision) {
>
> [region1]
>
> }
>
>
>
> Region coverage will tell you that whether [region1] was executed, and
> this will be enough to tell you that the branch decision evaluated to True
> at least once. However, this will not tell you how many times the decision
> evaluated to False, which is also important when considering branch
> coverage. In this case, you can calculate this value by subtracting the
> number of times [region0] was executed from the number of times [region1]
> was executed.
>
>
>
> What I am implementing for branch condition coverage will do this for you
> but will provide more granular True/False counts for individual conditions
> that comprise all Boolean expression (which also generate branches), and
> will tally all of these cases as branches that are either covered or not
> covered.
>
>
>
> I hope that helps.
>
>
>
> As for the future goals, I do plan on addressing what’s required for
> MC/DC, but that is work that I have not yet planned.
>
>
>
> Thanks!
>
>
>
> -Alan Phipps
>
>
>
> *From:* LoveZhi Xie [mailto:crazyforzhi1987 at gmail.com]
> *Sent:* Sunday, May 3, 2020 11:12 AM
> *To:* Phipps, Alan
> *Cc:* llvm-dev at lists.llvm.org
> *Subject:* Re: [EXTERNAL] [llvm-dev] How to get branch coverage by using
> 'source-based code coverage'
>
>
>
> Hi, Alan
>
>
>
> Really very excited to receive your email and sorry to be slow replying,
> it has been exceptionally busy over the last few days ;(
>
>
>
> Your explanation made the problem clear to me. So gcov branch coverage
> should be called condition coverage and clang region coverage
>
> is branch coverage in fact(also known as *decision/C1*), right?
>
>
>
> And llvm/clang will support all the following coverage criteria in future
>
> - Line coverage
> - Function coverage
> - Branch coverage/region coverage
> - Condition coverage
> - Modified Condition/Decision Coverage
>
> If there is any misunderstanding here, please point it out.
>
>
>
> Thanks,
>
> Ted Xie
>
>
>
>
>
>
>
> Phipps, Alan <a-phipps at ti.com> 于2020年4月28日周二 下午10:14写道：
>
> Hi Ted,
>
>
>
> Presently, branch coverage is not supported in clang source-based code
> coverage, but I am working on implementing branch condition coverage right
> now. It will track True/False branches for each leaf-level condition,
> including within Boolean expressions comprised of logical operators (“&&”,
> “||”). Technically, this condition-based coverage is more granular than
> how “branch coverage” is often defined (tracking on that control flow
> decisions allow for all regions to be hit), and is closer to the
> granularity that GCOV provides, but it’s anchored to the source code, won’t
> count “hidden branches”, and isn’t negatively impacted by optimization
> (although the reverse isn’t true: coverage instrumentation may impact
> optimization).
>
>
>
> My work is a step closer to getting us toward Modified Condition/Decision
> Coverage (MC/DC). I hope to upstream my work within the next few months,
> so unfortunately it won’t help you in the short-term. Region coverage may
> be enough for what you need if you can ascertain based on the coverage that
> control-flow decisions have evaluated to both true and false, but I’ll let
> others comment here with suggestions.
>
>
>
> -Alan Phipps
>
>
>
> *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] *On Behalf Of *LoveZhi
> Xie via llvm-dev
> *Sent:* Sunday, April 26, 2020 11:36 AM
> *To:* llvm-dev at lists.llvm.org
> *Subject:* [EXTERNAL] [llvm-dev] How to get branch coverage by using
> 'source-based code coverage'
>
>
>
> Hi, llvm/clang experts
>
>
>
> I need to get the branch coverage for some testing code. But i found gcov
> can't give a expected coverage which may
>
> count some 'hidden branch' in (See stackoverflow answer
> <https://stackoverflow.com/questions/42003783/lcov-gcov-branch-coverage-with-c-producing-branches-all-over-the-place>).
> Instead, I turn to use clang and the 'source-based code coverage' feature
>
> may be a right choice. But i can't find anything to describe branch
> coverage explicitly on the official site
> <https://clang.llvm.org/docs/SourceBasedCodeCoverage.html>.
>
>
>
> So how to use clang to get branch coverage by utilizing 'source-based...'
> feature? And is *region coverage *equivalent to branch coverage?
>
> (I tested on a simple program and region coverage seems pretty close to
> branch coverage)
>
>
>
> Any help is highly appreciated
>
>
>
> Thanks,
>
> Ted Xie
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200504/c241fdc0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cov1.png
Type: image/png
Size: 12113 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200504/c241fdc0/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cov2.png
Type: image/png
Size: 25337 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200504/c241fdc0/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cov3.png
Type: image/png
Size: 18297 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200504/c241fdc0/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cov4.png
Type: image/png
Size: 24148 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200504/c241fdc0/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cov5.png
Type: image/png
Size: 92921 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200504/c241fdc0/attachment-0009.png>