<div dir="ltr">Hi, Alan<div><br></div><div>Thanks for making it clear. But I was more confused now :(</div><div><br></div><div>I tested on a simple program and used both gcov and lcov to get branch coverage.</div><div>The code and build commands as below:</div><div><br></div><div><u><i><b>Example simple.cc</b></i></u></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font color="#0b5394" face="trebuchet ms, sans-serif">#include <string></font></div><div><font color="#0b5394" face="trebuchet ms, sans-serif">// If not comment this line, the branch coverage won't reach to 100%</font></div><div><font color="#0b5394" face="trebuchet ms, sans-serif">// #include <iostream></font></div><div><font color="#0b5394" face="trebuchet ms, sans-serif">int main(int argc, const char* argv[]) {</font></div><div><font color="#0b5394" face="trebuchet ms, sans-serif"> std::string str = "foo";</font></div><div><font color="#0b5394" face="trebuchet ms, sans-serif"> str = argv[1];</font></div><div><font color="#0b5394" face="trebuchet ms, sans-serif"> if (str == "foo" || str == "bar") {</font></div><div><font color="#0b5394" face="trebuchet ms, sans-serif"> int a = 5;</font></div><div><font color="#0b5394" face="trebuchet ms, sans-serif"> a = 7;</font></div><div><font color="#0b5394" face="trebuchet ms, sans-serif"> } else {</font></div><div><font color="#0b5394" face="trebuchet ms, sans-serif"> int b = 6;</font></div><div><font color="#0b5394" face="trebuchet ms, sans-serif"> b = 4;</font></div><div><font color="#0b5394" face="trebuchet ms, sans-serif"> }</font></div><div><font color="#0b5394" face="trebuchet ms, sans-serif"> return 0;</font></div><div><font color="#0b5394" face="trebuchet ms, sans-serif">}</font></div></blockquote><div><font face="trebuchet ms, sans-serif" color="#000000"><b style=""><i style=""><br></i></b></font></div><div><font face="trebuchet ms, sans-serif" color="#000000"><b style=""><i style=""><u>Coverage build commands</u></i></b></font></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font face="comic sans ms, sans-serif" color="#0b5394">#!/bin/bash</font></div><div><font face="comic sans ms, sans-serif" color="#0b5394">g++ -o simple -fno-exceptions -fno-inline -fprofile-arcs -ftest-coverage simple.cc</font></div><div><font face="comic sans ms, sans-serif" color="#0b5394">./simple "foo"</font></div><div><font face="comic sans ms, sans-serif" color="#0b5394">./simple "bar"</font></div><div><font face="comic sans ms, sans-serif" color="#0b5394">./simple "hello1"</font></div><div><font face="comic sans ms, sans-serif" color="#0b5394">./simple "hello2"</font></div><div><font face="comic sans ms, sans-serif" color="#0b5394">./simple "hello3"</font></div><div><font face="comic sans ms, sans-serif" color="#0b5394">./simple "hello4"</font></div><div><font face="comic sans ms, sans-serif" color="#0b5394">./simple "hello5"</font></div><div><font face="comic sans ms, sans-serif" color="#0b5394">./simple "hello6"</font></div><div><font face="comic sans ms, sans-serif" color="#0b5394">./simple "hello7"</font></div><div><font face="comic sans ms, sans-serif" color="#0b5394">./simple "hello8"</font></div><div><font face="comic sans ms, sans-serif" color="#0b5394">lcov --rc lcov_branch_coverage=1 --capture --directory . --output-file <a href="http://simple.lcov.info">simple.lcov.info</a></font></div><div><font face="comic sans ms, sans-serif" color="#0b5394">genhtml <a href="http://simple.lcov.info">simple.lcov.info</a> --function-coverage --branch-coverage --output-directory lcov_out/</font></div><div><font face="comic sans ms, sans-serif" color="#0b5394">gcov -b -c simple.gcno</font></div><div><br></div></blockquote><br><div>Both gcov and lcov will give a 100% branch coverage report:</div><div><div><img src="cid:ii_k9s653m80" alt="cov1.png" width="237" height="87" style="margin-right: 0px;"> <img src="cid:ii_k9s65idv1" alt="cov2.png" width="414" height="67" style="margin-right: 0px;"></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><br></div></blockquote><div>My questions:</div><div><ul><li> What's the "<font color="#0b5394"><b>official</b></font>" evaluation formula on branch coverage?</li></ul><div> Sorry I can't find any compelling documents to describe the accurate algorithm on this issue.</div><div> I just thought gcov/lcov won't consider <span style="color:rgb(31,73,125);font-family:Calibri,sans-serif;font-size:14.6667px"><i>how many times the decision evaluated to False.</i></span></div><div><span style="color:rgb(31,73,125);font-family:Calibri,sans-serif;font-size:14.6667px"><i> P</i></span>lease see the gcov file on my example: simple.cc. There are 6 branches in total which are all hit</div><div> So the result would be 6/6 * 100% = 100%</div><div><span style="color:rgb(31,73,125);font-family:Calibri,sans-serif;font-size:14.6667px"><i> </i></span></div><div> <img src="cid:ii_k9s6vwfa3" alt="cov3.png" width="418" height="235" style="margin-right: 0px;"><br></div><div> </div><div><ul><li>Are there any existing post-processing tools based on clang/llvm source based coverage?</li></ul></div></div><div> I just saw a <a href="https://github.com/zero9178/C-Cpp-Coverage-for-CLion">clion plugin</a> at github which may generate branch coverage by using region coverage data. But this depends on clion</div><div> which is not easy to implement automation test.</div><div><ul><li>How to disable STL (or other) noise in gcov branch coverage?</li></ul><div> Sorry, this is not clang/llvm question. But I am still looking forward to some suggestions. For example, if I add iostream header back to</div></div><div> simple.cc, then we won't get a 100% branch coverage. This may because iostream will introduce some global destruction branch.</div><div> </div><div> <img src="cid:ii_k9s7fz6t4" alt="cov4.png" width="472" height="94"><br></div><div><br></div><div> Or if it's impossible, can we think clang region coverage should be a best practice <i>in C++ testing world</i>??</div><div> </div><div> AFAIK, chromium is using region coverage now.</div><div> <img src="cid:ii_k9s7lnet5" alt="cov5.png" width="472" height="145"><br></div><div><br></div><div>FYI.</div><div><br></div><div>Thanks,</div><div>Ted Xie</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Phipps, Alan <<a href="mailto:a-phipps@ti.com">a-phipps@ti.com</a>> 于2020年5月4日周一 上午12:54写道：<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_7189132425558485252WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">+ Vedant Kumar<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hi Ted!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">I would not say that clang region coverage is branch coverage(‘aka decision coverage’). I would say that it may be possible to extrapolate branch decision
coverage based on region coverage. For example, consider the following:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:Consolas;color:rgb(31,73,125)">[region0]<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:Consolas;color:rgb(31,73,125)">if (decision) {<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:Consolas;color:rgb(31,73,125)"> [region1]<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:0.5in"><span style="font-size:11pt;font-family:Consolas;color:rgb(31,73,125)">}<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Region coverage will tell you that whether [region1] was executed, and this will be enough to tell you that the branch decision evaluated to True at least once.
However, this will not tell you how many times the decision evaluated to False, which is also important when considering branch coverage. In this case, you can calculate this value by subtracting the number of times [region0] was executed from the number
of times [region1] was executed.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">What I am implementing for branch condition coverage will do this for you but will provide more granular True/False counts for individual conditions that comprise
all Boolean expression (which also generate branches), and will tally all of these cases as branches that are either covered or not covered.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">I hope that helps.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">As for the future goals, I do plan on addressing what’s required for MC/DC, but that is work that I have not yet planned.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-Alan Phipps<u></u><u></u></span></p>
<p class="MsoNormal"><a name="m_7189132425558485252__MailEndCompose"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></a></p>
<p class="MsoNormal"><b><span style="font-size:10pt;font-family:Tahoma,sans-serif">From:</span></b><span style="font-size:10pt;font-family:Tahoma,sans-serif"> LoveZhi Xie [mailto:<a href="mailto:crazyforzhi1987@gmail.com" target="_blank">crazyforzhi1987@gmail.com</a>]
<br>
<b>Sent:</b> Sunday, May 3, 2020 11:12 AM<br>
<b>To:</b> Phipps, Alan<br>
<b>Cc:</b> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<b>Subject:</b> Re: [EXTERNAL] [llvm-dev] How to get branch coverage by using 'source-based code coverage'<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi, Alan<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Really very excited to receive your email and sorry to be slow replying, it has been exceptionally busy over the last few days ;(<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Your explanation made the problem clear to me. So gcov branch coverage should be called condition coverage and clang region coverage<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">is branch coverage in fact(also known as <b><span style="color:rgb(39,78,19)">decision/C1</span></b><span style="color:black">)</span>, right?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">And llvm/clang will support all the following coverage criteria in future<u></u><u></u></p>
</div>
<div>
<ul type="disc">
<li class="MsoNormal">
Line coverage<u></u><u></u></li><li class="MsoNormal">
Function coverage<u></u><u></u></li><li class="MsoNormal">
Branch coverage/region coverage<u></u><u></u></li><li class="MsoNormal">
Condition coverage<u></u><u></u></li><li class="MsoNormal">
Modified Condition/Decision Coverage<u></u><u></u></li></ul>
<div>
<p class="MsoNormal">If there is any misunderstanding here, please point it out.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal">Ted Xie<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">Phipps, Alan <<a href="mailto:a-phipps@ti.com" target="_blank">a-phipps@ti.com</a>>
<span style="font-family:"MS Gothic"">于</span>2020<span style="font-family:"MS Gothic"">年</span>4<span style="font-family:"MS Gothic"">月</span>28<span style="font-family:"MS Gothic"">日周二</span>
<span style="font-family:"MS Gothic"">下午</span>10:14<span style="font-family:"MS Gothic"">写道：</span><u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hi Ted,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Presently, branch coverage is not supported in clang source-based code coverage, but I am working
on implementing branch condition coverage right now. It will track True/False branches for each leaf-level condition, including within Boolean expressions comprised of logical operators (“&&”, “||”). Technically, this condition-based coverage is more granular
than how “branch coverage” is often defined (tracking on that control flow decisions allow for all regions to be hit), and is closer to the granularity that GCOV provides, but it’s anchored to the source code, won’t count “hidden branches”, and isn’t negatively
impacted by optimization (although the reverse isn’t true: coverage instrumentation may impact optimization).</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">My work is a step closer to getting us toward Modified Condition/Decision Coverage (MC/DC). I hope
to upstream my work within the next few months, so unfortunately it won’t help you in the short-term. Region coverage may be enough for what you need if you can ascertain based on the coverage that control-flow decisions have evaluated to both true and false,
but I’ll let others comment here with suggestions.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-Alan Phipps</span><u></u><u></u></p>
<p class="MsoNormal"><a name="m_7189132425558485252_m_1180875836774759463__MailEndCompose"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span></a><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:10pt;font-family:Tahoma,sans-serif">From:</span></b><span style="font-size:10pt;font-family:Tahoma,sans-serif"> llvm-dev [mailto:<a href="mailto:llvm-dev-bounces@lists.llvm.org" target="_blank">llvm-dev-bounces@lists.llvm.org</a>]
<b>On Behalf Of </b>LoveZhi Xie via llvm-dev<br>
<b>Sent:</b> Sunday, April 26, 2020 11:36 AM<br>
<b>To:</b> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<b>Subject:</b> [EXTERNAL] [llvm-dev] How to get branch coverage by using 'source-based code coverage'</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">Hi, llvm/clang experts<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I need to get the branch coverage for some testing code. But i found gcov can't give a expected coverage which may<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">count some 'hidden branch' in (See <a href="https://stackoverflow.com/questions/42003783/lcov-gcov-branch-coverage-with-c-producing-branches-all-over-the-place" target="_blank">stackoverflow
answer</a>). Instead, I turn to use clang and the 'source-based code coverage' feature<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">may be a right choice. But i can't find anything to describe branch coverage explicitly on the
<a href="https://clang.llvm.org/docs/SourceBasedCodeCoverage.html" target="_blank">
official site</a>. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">So how to use clang to get branch coverage by utilizing 'source-based...' feature? And is
<b><i>region coverage </i></b>equivalent to branch coverage?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">(I tested on a simple program and region coverage seems pretty close to branch coverage)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Any help is highly appreciated<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Ted Xie<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote></div>