<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<p>Thanks, Alan. This certainly seems useful. Can you please provide a quick overview on how this relates to our other infrastructure for coverage, for profiling, and what's used for fuzz testing?</p>
<p> -Hal<br>
</p>
<div class="moz-cite-prefix">On 1/23/20 6:09 PM, Phipps, Alan via llvm-dev wrote:<br>
</div>
<blockquote type="cite" cite="mid:a90baf3ed0a146d49b1e39cfcff75005@ti.com">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1066956857;
mso-list-type:hybrid;
mso-list-template-ids:1692585554 479752080 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:3;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1
{mso-list-id:1258447642;
mso-list-type:hybrid;
mso-list-template-ids:1349780036 -1128909910 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l1:level1
{mso-level-text:"%1\.\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Vedant Kumar asked me to post my design thoughts concerning branch coverage at llvm-dev since there is general interest.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My team at Texas Instruments is developing an embedded ARM C/C++ compiler with LLVM. I would like to enhance LLVM’s code coverage capability with branch condition coverage (for C/C++), similar to GCC/GCOV support for branch coverage.
This is useful for TI, and I think this will be a useful feature enhancement to LLVM that I can upstream.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In a nutshell, the functionality boils down to tracking how many times a generated “branch” instruction (based on a source code condition) is taken or not taken (i.e. evaluated into “True” and “False”). This applies to decision points
in control flow (if, for, while, …) as well as individual conditions on logical operators (“&&”, “||”) in Boolean expressions.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In sketching out a design, there are three primary areas in the design that I am proposing:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo1"><!--[if !supportLists]--><b><span style="mso-list:Ignore">1.)<span style="font:7.0pt
"Times New Roman"">
</span></span></b><!--[endif]--><b>Add a new CounterMappingRegion kind for branch conditions<o:p></o:p></b></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l1 level2
lfo1">
<!--[if !supportLists]--><span style="mso-list:Ignore">a.<span style="font:7.0pt "Times New Roman"">
</span></span><!--[endif]-->This new region kind would track two counters, one for the “True” branch taken count of a branch condition, and one for the “False” branch taken count.<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.5in;text-indent:-1.5in;mso-text-indent-alt:-9.0pt;mso-list:l1
level3 lfo1">
<!--[if !supportLists]--><span style="mso-list:Ignore"><span style="font:7.0pt "Times New Roman"">
</span>i.<span style="font:7.0pt "Times New
Roman""> </span></span><!--[endif]-->Alternatively, I could use two separate CounterMappingRegions to track individual counters since this is how the class was originally written to be used. However,
using a single region kind to represent a single branch condition that ties all of the pertinent counter information together seems like a cleaner design.<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.5in;text-indent:-1.5in;mso-text-indent-alt:-9.0pt;mso-list:l1
level3 lfo1">
<!--[if !supportLists]--><span style="mso-list:Ignore"><span style="font:7.0pt "Times New Roman"">
</span>ii.<span style="font:7.0pt "Times New
Roman""> </span></span><!--[endif]-->Just as for all counters, the two branch condition counters can represent a reference to an instrumentation counter or to a counter expression. The two counters
are encoded along with the MappingRegions and distinguished based on the region kind.<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.5in;text-indent:-1.5in;mso-text-indent-alt:-9.0pt;mso-list:l1
level3 lfo1">
<!--[if !supportLists]--><span style="mso-list:Ignore"><span style="font:7.0pt "Times New Roman"">
</span>iii.<span style="font:7.0pt "Times New
Roman""> </span>
</span><!--[endif]-->All other CounterMappingRegion kinds simply ignore the second counter; nothing changes in how they’re encoded, which preserves format backward compatibility.<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l1 level2
lfo1">
<!--[if !supportLists]--><span style="mso-list:Ignore">b.<span style="font:7.0pt "Times New Roman"">
</span></span><!--[endif]-->I think this change also requires an adjustment to the class SourceMappingRegion to support branch conditions that can be generated into CounterMappingRegion instances.<o:p></o:p></p>
<p class="MsoListParagraph"><o:p> </o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo1"><!--[if !supportLists]--><b><span style="mso-list:Ignore">2.)<span style="font:7.0pt
"Times New Roman"">
</span></span></b><!--[endif]--><b>Counter Instrumentation<o:p></o:p></b></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l1 level2
lfo1">
<!--[if !supportLists]--><span style="mso-list:Ignore">a.<span style="font:7.0pt "Times New Roman"">
</span></span><!--[endif]-->We can reuse most of the existing profile instrumentation counters that are emitted as part of profiling/coverage to calculate branch condition counts (True/False).
<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.5in;text-indent:-1.5in;mso-text-indent-alt:-9.0pt;mso-list:l1
level3 lfo1">
<!--[if !supportLists]--><span style="mso-list:Ignore"><span style="font:7.0pt "Times New Roman"">
</span>i.<span style="font:7.0pt "Times New
Roman""> </span></span><!--[endif]-->This assumption leverages the fact that logical operators
<i>in C</i> are “short-circuit” operators. For example, the “False-taken” count for the left-hand-side condition in a
<i>logical-or</i> expression (e.g. condition “C1” in “C1 || C2”) can be derived from the execution count we already track for the right-hand-side (condition “C2” in “C1 || C2”).<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l1 level2
lfo1">
<!--[if !supportLists]--><span style="mso-list:Ignore">b.<span style="font:7.0pt "Times New Roman"">
</span></span><!--[endif]-->There does exist a case when evaluating the right-hand-side condition of a logical operator
<i>that isn’t part of a control-flow statement</i> (e.g. condition “C2” in “x = C1 || C2;”) that will require instrumenting a new counter in order to properly derive that condition’s “true” count and “false” count.<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l1 level2
lfo1">
<!--[if !supportLists]--><span style="mso-list:Ignore">c.<span style="font:7.0pt "Times New Roman"">
</span></span><!--[endif]-->I’ll avoid going too deep into detail here, but my goal is to ensure we reuse existing profile counters as much as possible.<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in"><o:p> </o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo1"><!--[if !supportLists]--><b><span style="mso-list:Ignore">3.)<span style="font:7.0pt
"Times New Roman"">
</span></span></b><!--[endif]--><b>Visualization using llvm-cov<o:p></o:p></b></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l1 level2
lfo1">
<!--[if !supportLists]--><span style="mso-list:Ignore">a.<span style="font:7.0pt "Times New Roman"">
</span></span><!--[endif]-->The notion of CoverageSegment needs to be extended to comprehend the branch condition data represented by a CounterMappingRegion above. But then llvm-cov can treat the segment distinctly when displaying True/False counts for each
branch condition as well as tracking total missed branches.<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l1 level2
lfo1">
<!--[if !supportLists]--><span style="mso-list:Ignore">b.<span style="font:7.0pt "Times New Roman"">
</span></span><!--[endif]-->We can also add a BranchCoverageInfo class to track branch coverage data, similar to LineCoverageInfo and RegionCoverageInfo.<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l1 level2
lfo1">
<!--[if !supportLists]--><span style="mso-list:Ignore">c.<span style="font:7.0pt "Times New Roman"">
</span></span><!--[endif]-->The text output could look something like GCOV but with more detail that we know (I prototyped this using logical-or):<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:0in"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:Consolas"> 9| |int main(int argc, char *argv[])</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:Consolas"> 10| 3|{</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:Consolas"> 11| 3| if (argc == 1)</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><b><span style="font-size:10.0pt;font-family:Consolas">Branch (11:9): [True: 1, False: 2]</span></b><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:Consolas"> 12| 1| {</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:Consolas"> 13| 1| return 0;</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:Consolas"> 14| 1| }</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:.5in"><span style="font-size:10.0pt;font-family:Consolas">. . .</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:Consolas"> 23| 2| if (a == 0 || b == 2 || b == 34 || a == b)</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><b><span style="font-size:10.0pt;font-family:Consolas">Branch (23:9): [True: 1, False: 1]</span></b><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><b><span style="font-size:10.0pt;font-family:Consolas">Branch (23:19): [True: 1, False: 0]</span></b><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><b><span style="font-size:10.0pt;font-family:Consolas">Branch (23:29): [True: 0, False: 0]</span></b><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><b><span style="font-size:10.0pt;font-family:Consolas">Branch (23:40): [True: 0, False: 0]</span></b><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in;text-indent:.5in"><span style="font-size:10.0pt;font-family:Consolas">. . .</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:Consolas"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><span style="font-size:10.0pt;font-family:Consolas"> 31| 2| b = a || c; </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><b><span style="font-size:10.0pt;font-family:Consolas">Branch (31:9): [True: 1, False: 1]</span></b><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:1.5in"><b><span style="font-size:10.0pt;font-family:Consolas">Branch (31:14): [True: 1, False: 0]</span></b><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l1 level2
lfo1">
<!--[if !supportLists]--><span style="mso-list:Ignore">d.<span style="font:7.0pt "Times New Roman"">
</span></span><!--[endif]-->I thought about extending the “region-count” carat markers in the text display, but it could get messy. For the HTML output, we can get a bit more fancy.<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l1 level2
lfo1">
<!--[if !supportLists]--><span style="mso-list:Ignore">e.<span style="font:7.0pt "Times New Roman"">
</span></span><!--[endif]-->Branch miss percentages/totals will be added to the coverage report.<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in"><o:p> </o:p></p>
<p class="MsoListParagraph" style="margin-left:0in"><b>Additional Notes<o:p></o:p></b></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo2"><!--[if !supportLists]--><span style="mso-list:Ignore">-<span style="font:7.0pt "Times
New Roman"">
</span></span><!--[endif]-->I’m aware that constant condition folding in CodeGenFunction::EmitBranchOnBoolExpr() needs to be taken into account. Is there anything else related to branch optimization that I ought to be aware of?<b><o:p></o:p></b></p>
<p class="MsoListParagraph"><b><o:p> </o:p></b></p>
<p class="MsoListParagraph" style="margin-left:0in"><o:p> </o:p></p>
<p class="MsoListParagraph" style="margin-left:0in">Please let me know if these design thoughts look reasonable and if this would be useful. The goal is to start full implementation soon and upstream in a few months.<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:0in"><o:p> </o:p></p>
<p class="MsoListParagraph" style="margin-left:0in">Thanks!<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:0in">Alan Phipps<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:0in">Texas Instruments, Inc.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
LLVM Developers mailing list
<a class="moz-txt-link-abbreviated" href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>
<a class="moz-txt-link-freetext" href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory</pre>
</body>
</html>