[llvm-dev] [EXTERNAL] How to get branch coverage by using 'source-based code coverage'
LoveZhi Xie via llvm-dev
llvm-dev at lists.llvm.org
Wed May 6 09:36:23 PDT 2020
Hi, Alan
Many thanks to your answer. I mistook your meaning because I just noticed
you said "times" on coverage.
I tried the clang on* 'if-only'* code just now and you're right. The actual
result was certainly not branch coverage.
I just expect to see your work be merged soon. 👍
Thanks,
Ted Xie
Phipps, Alan <a-phipps at ti.com> 于2020年5月7日周四 上午12:03写道：
> Hi Ted,
>
>
>
> I’ll attempt to answer your questions:
>
> 1.) What's the "*official*" evaluation formula on branch coverage?
> Sorry I can't find any compelling documents to describe the accurate
> algorithm on this issue. I just thought gcov/lcov won't consider *how
> many times the decision evaluated to False.*
>
> *a. *[Alan]: I share your concern here. My reference is the ISO26262
> standard for functional safety, which defines branch coverage as “*percentage
> of branches of the control flow of a computer program executed during a
> test. Note 1: 100% branch coverage implies 100% statement coverage., Note
> 2: An if-statement always has two branches – condition true and condition
> false – independent of the existence of an else-clause*”
>
> * i. *Basically,
> condition *true* and condition *false* for every decision in a program.
> In contrast, MC/DC applies to *single condition outcomes* that affect a
> decision.
>
> *b. *[Alan] Your GCOV output is producing branch information for
> each condition of the decision, and each of those branches corresponds to
> the true and false evaluation of each condition. The problem with GCC/GCOV
> is that, because of the way it instruments, it’s not obvious how branches
> correspond to the source code.
>
>
>
> 2.) Are there any existing post-processing tools based on clang/llvm
> source based coverage
>
> a. [Alan] I’m not aware of them, but others may know
>
>
>
> 3.) How to disable STL (or other) noise in gcov branch coverage?
>
> a. [Alan] I think this is a perennial problem – gcov reports hidden
> branches that are created, but the data isn’t necessarily useful from a
> source-code coverage perspective.
>
>
>
> -Alan
>
>
>
> *From:* LoveZhi Xie [mailto:crazyforzhi1987 at gmail.com]
> *Sent:* Monday, May 4, 2020 3:24 AM
> *To:* Phipps, Alan
> *Cc:* llvm-dev at lists.llvm.org; Vedant Kumar
> *Subject:* Re: [EXTERNAL] [llvm-dev] How to get branch coverage by using
> 'source-based code coverage'
>
>
>
> Hi, Alan
>
>
>
> Thanks for making it clear. But I was more confused now :(
>
>
>
> I tested on a simple program and used both gcov and lcov to get branch
> coverage.
>
> The code and build commands as below:
>
>
>
> *Example simple.cc*
>
> #include <string>
>
> // If not comment this line, the branch coverage won't reach to 100%
>
> // #include <iostream>
>
> int main(int argc, const char* argv[]) {
>
> std::string str = "foo";
>
> str = argv[1];
>
> if (str == "foo" || str == "bar") {
>
> int a = 5;
>
> a = 7;
>
> } else {
>
> int b = 6;
>
> b = 4;
>
> }
>
> return 0;
>
> }
>
>
>
> *Coverage build commands*
>
> #!/bin/bash
>
> g++ -o simple -fno-exceptions -fno-inline -fprofile-arcs -ftest-coverage
> simple.cc
>
> ./simple "foo"
>
> ./simple "bar"
>
> ./simple "hello1"
>
> ./simple "hello2"
>
> ./simple "hello3"
>
> ./simple "hello4"
>
> ./simple "hello5"
>
> ./simple "hello6"
>
> ./simple "hello7"
>
> ./simple "hello8"
>
> lcov --rc lcov_branch_coverage=1 --capture --directory . --output-file
> simple.lcov.info
>
> genhtml simple.lcov.info --function-coverage --branch-coverage
> --output-directory lcov_out/
>
> gcov -b -c simple.gcno
>
>
>
>
>
> Both gcov and lcov will give a 100% branch coverage report:
>
> [image: cov1.png] [image: cov2.png]
>
>
>
> My questions:
>
> - What's the "*official*" evaluation formula on branch coverage?
>
> Sorry I can't find any compelling documents to describe the
> accurate algorithm on this issue.
>
> I just thought gcov/lcov won't consider *how many times the
> decision evaluated to False.*
>
> * P*lease see the gcov file on my example: simple.cc. There are
> 6 branches in total which are all hit
>
> So the result would be 6/6 * 100% = 100%
>
>
>
> [image: cov3.png]
>
>
>
> - Are there any existing post-processing tools based on clang/llvm
> source based coverage?
>
> I just saw a clion plugin
> <https://github.com/zero9178/C-Cpp-Coverage-for-CLion> at github which
> may generate branch coverage by using region coverage data. But this
> depends on clion
>
> which is not easy to implement automation test.
>
> - How to disable STL (or other) noise in gcov branch coverage?
>
> Sorry, this is not clang/llvm question. But I am still looking
> forward to some suggestions. For example, if I add iostream header back to
>
> simple.cc, then we won't get a 100% branch coverage. This may
> because iostream will introduce some global destruction branch.
>
>
>
> [image: cov4.png]
>
>
>
> Or if it's impossible, can we think clang region coverage
> should be a best practice *in C++ testing world*??
>
> AFAIK, chromium is using region coverage now.
>
> [image: cov5.png]
>
>
>
> FYI.
>
>
>
> Thanks,
>
> Ted Xie
>
>
>
> Phipps, Alan <a-phipps at ti.com> 于2020年5月4日周一 上午12:54写道：
>
> + Vedant Kumar
>
>
>
> Hi Ted!
>
>
>
> I would not say that clang region coverage is branch coverage(‘aka
> decision coverage’). I would say that it may be possible to extrapolate
> branch decision coverage based on region coverage. For example, consider
> the following:
>
>
>
> [region0]
>
> if (decision) {
>
> [region1]
>
> }
>
>
>
> Region coverage will tell you that whether [region1] was executed, and
> this will be enough to tell you that the branch decision evaluated to True
> at least once. However, this will not tell you how many times the decision
> evaluated to False, which is also important when considering branch
> coverage. In this case, you can calculate this value by subtracting the
> number of times [region0] was executed from the number of times [region1]
> was executed.
>
>
>
> What I am implementing for branch condition coverage will do this for you
> but will provide more granular True/False counts for individual conditions
> that comprise all Boolean expression (which also generate branches), and
> will tally all of these cases as branches that are either covered or not
> covered.
>
>
>
> I hope that helps.
>
>
>
> As for the future goals, I do plan on addressing what’s required for
> MC/DC, but that is work that I have not yet planned.
>
>
>
> Thanks!
>
>
>
> -Alan Phipps
>
>
>
> *From:* LoveZhi Xie [mailto:crazyforzhi1987 at gmail.com]
> *Sent:* Sunday, May 3, 2020 11:12 AM
> *To:* Phipps, Alan
> *Cc:* llvm-dev at lists.llvm.org
> *Subject:* Re: [EXTERNAL] [llvm-dev] How to get branch coverage by using
> 'source-based code coverage'
>
>
>
> Hi, Alan
>
>
>
> Really very excited to receive your email and sorry to be slow replying,
> it has been exceptionally busy over the last few days ;(
>
>
>
> Your explanation made the problem clear to me. So gcov branch coverage
> should be called condition coverage and clang region coverage
>
> is branch coverage in fact(also known as *decision/C1*), right?
>
>
>
> And llvm/clang will support all the following coverage criteria in future
>
> - Line coverage
> - Function coverage
> - Branch coverage/region coverage
> - Condition coverage
> - Modified Condition/Decision Coverage
>
> If there is any misunderstanding here, please point it out.
>
>
>
> Thanks,
>
> Ted Xie
>
>
>
>
>
>
>
> Phipps, Alan <a-phipps at ti.com> 于2020年4月28日周二 下午10:14写道：
>
> Hi Ted,
>
>
>
> Presently, branch coverage is not supported in clang source-based code
> coverage, but I am working on implementing branch condition coverage right
> now. It will track True/False branches for each leaf-level condition,
> including within Boolean expressions comprised of logical operators (“&&”,
> “||”). Technically, this condition-based coverage is more granular than
> how “branch coverage” is often defined (tracking on that control flow
> decisions allow for all regions to be hit), and is closer to the
> granularity that GCOV provides, but it’s anchored to the source code, won’t
> count “hidden branches”, and isn’t negatively impacted by optimization
> (although the reverse isn’t true: coverage instrumentation may impact
> optimization).
>
>
>
> My work is a step closer to getting us toward Modified Condition/Decision
> Coverage (MC/DC). I hope to upstream my work within the next few months,
> so unfortunately it won’t help you in the short-term. Region coverage may
> be enough for what you need if you can ascertain based on the coverage that
> control-flow decisions have evaluated to both true and false, but I’ll let
> others comment here with suggestions.
>
>
>
> -Alan Phipps
>
>
>
> *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] *On Behalf Of *LoveZhi
> Xie via llvm-dev
> *Sent:* Sunday, April 26, 2020 11:36 AM
> *To:* llvm-dev at lists.llvm.org
> *Subject:* [EXTERNAL] [llvm-dev] How to get branch coverage by using
> 'source-based code coverage'
>
>
>
> Hi, llvm/clang experts
>
>
>
> I need to get the branch coverage for some testing code. But i found gcov
> can't give a expected coverage which may
>
> count some 'hidden branch' in (See stackoverflow answer
> <https://stackoverflow.com/questions/42003783/lcov-gcov-branch-coverage-with-c-producing-branches-all-over-the-place>).
> Instead, I turn to use clang and the 'source-based code coverage' feature
>
> may be a right choice. But i can't find anything to describe branch
> coverage explicitly on the official site
> <https://clang.llvm.org/docs/SourceBasedCodeCoverage.html>.
>
>
>
> So how to use clang to get branch coverage by utilizing 'source-based...'
> feature? And is *region coverage *equivalent to branch coverage?
>
> (I tested on a simple program and region coverage seems pretty close to
> branch coverage)
>
>
>
> Any help is highly appreciated
>
>
>
> Thanks,
>
> Ted Xie
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200507/36499cfe/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 33987 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200507/36499cfe/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 18185 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200507/36499cfe/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.jpg
Type: image/jpeg
Size: 10451 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200507/36499cfe/attachment-0003.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image008.jpg
Type: image/jpeg
Size: 6144 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200507/36499cfe/attachment-0004.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image010.jpg
Type: image/jpeg
Size: 13518 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200507/36499cfe/attachment-0005.jpg>
More information about the llvm-dev
mailing list