[PATCH] D84467: Add support for Branch Coverage in LLVM Source-Based Code Coverage

Thu Jul 23 14:31:16 PDT 2020

alanphipps created this revision.
alanphipps added reviewers: vsk, rnk, hans, efriedma.
alanphipps added a project: clang.
Herald added subscribers: llvm-commits, cfe-commits, mgrang, hiraditya.
Herald added a project: LLVM.

In January, I posted a basic design outline on llvm-dev for supporting Branch Coverage as part of LLVM Source-based Code Coverage.  In general, I think this is a straightforward extension of source-based code coverage to measure True/False execution counts of branch-generating conditions in the source code.

Note that by "branch coverage", I am referring to a //granular // level of measurement for each leaf-level boolean expression (i.e. "condition") that may compose larger boolean expressions using logical operators.  This is an equivalent level of granularity to what is reported by GCOV.   This granular level of branch coverage I implemented is amenable to further extension for MC/DC analysis.  I am not referring to higher-level "decision coverage".

I apologize for the lengthy review -- there are three main sections, but they're interconnected.  Most of the other changes are for testing (which, I confirmed, is sufficient to provide coverage of everything that was added).  This is my first phabricator review, so please give me suggestions or improvement!

Examples of output for Text and HTML:

Text: f{F12395372 <https://reviews.llvm.org/F12395372>}
HTML (as PDF): F12395445: demo.c.html.pdf <https://reviews.llvm.org/F12395445>

Three main chunks of work:

- Adding and using "Branch Regions":

Extending the coverage mapping format with a notion of a "branch region" kind with an additional alternate "FalseCount" counter. Both counters can then be used  to measure "True" and "False" branch counts. Also extend the Writer/Reader encoding routines to comprehend it.

  clang/lib/CodeGen/CoverageMappingGen.cpp
  clang/lib/CodeGen/CoverageMappingGen.h
  llvm/include/llvm/ProfileData/Coverage/CoverageMapping.h
  llvm/lib/ProfileData/Coverage/CoverageMappingWriter.cpp
  llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp

- Instrumenting logical operators ("&&", "||"):

The existing counter instrumentation can be leveraged for most branch regions.  Logical operators are a bit special and require an additional counter.

Consider: "X = C1 || C2;"

For logical operators, an execution counter is already emitted to track the execution count for the right-hand-side of the expression (C2), and because of logical operator short-circuit semantics, this counter can tell you whether they left-hand-side (C1) evaluated to True or False (depending on whether it is logical-AND or logical-OR).  However, this doesn't present enough information to tell you how many times the right-hand-side (C2) evaluated to True or False.  Therefore, we need to instrument an additional counter here that is dependent on the operator semantics.

Note that there is room for optimization here: when a boolean expression with logical operators is used with a control-flow statement (like if-stmt), we can leverage the "Then" block counter.  And in cases where you have multiple levels of nested logical operators, we can re-use counters.  For this review, I have chosen to keep the design straightforward and will defer these optimizations to a future patch.

Since the new counters extend the profile format, I bumped the profile version to allow backward compatibility with older profile versions.

  clang/lib/CodeGen/CGExprScalar.cpp
  clang/lib/CodeGen/CGStmt.cpp
  clang/lib/CodeGen/CodeGenFunction.cpp
  clang/lib/CodeGen/CodeGenFunction.h
  clang/lib/CodeGen/CodeGenPGO.cpp
  llvm/include/llvm/ProfileData/InstrProf.h
  llvm/include/llvm/ProfileData/InstrProfData.inc

- Visualization (llvm-cov):

Extend the llvm-cov built-in notion of "SubView" to visualize branches for whole files, individual functions, and macro expansions (which are recursive because macros can be based on other macros). SubViews keep the visualization regions visually distinct.

Extend summary reports to include branch coverage
Enable branch coverage data to be exportable into JSON and LCOV formats.

  llvm/lib/ProfileData/Coverage/CoverageMapping.cpp
  llvm/tools/llvm-cov/CodeCoverage.cpp
  llvm/tools/llvm-cov/CoverageReport.cpp
  llvm/tools/llvm-cov/CoverageSummaryInfo.cpp
  llvm/tools/llvm-cov/CoverageSummaryInfo.h
  llvm/tools/llvm-cov/CoverageViewOptions.h
  llvm/tools/llvm-cov/SourceCoverageView.cpp
  llvm/tools/llvm-cov/SourceCoverageView.h
  llvm/tools/llvm-cov/SourceCoverageViewHTML.cpp
  llvm/tools/llvm-cov/SourceCoverageViewHTML.h
  llvm/tools/llvm-cov/SourceCoverageViewText.cpp
  llvm/tools/llvm-cov/SourceCoverageViewText.h
  llvm/tools/llvm-cov/CoverageExporterJson.cpp
  llvm/tools/llvm-cov/CoverageExporterLcov.cpp

Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D84467

Files:
  clang/lib/CodeGen/CGExprScalar.cpp
  clang/lib/CodeGen/CGStmt.cpp
  clang/lib/CodeGen/CodeGenFunction.cpp
  clang/lib/CodeGen/CodeGenFunction.h
  clang/lib/CodeGen/CodeGenPGO.cpp
  clang/lib/CodeGen/CoverageMappingGen.cpp
  clang/lib/CodeGen/CoverageMappingGen.h
  clang/test/CoverageMapping/branch-constfolded.cpp
  clang/test/CoverageMapping/branch-logical-mixed.cpp
  clang/test/CoverageMapping/branch-macros.cpp
  clang/test/CoverageMapping/branch-mincounters.cpp
  clang/test/CoverageMapping/branch-templates.cpp
  clang/test/CoverageMapping/continue.c
  clang/test/CoverageMapping/coroutine.cpp
  clang/test/CoverageMapping/if.cpp
  clang/test/CoverageMapping/label.cpp
  clang/test/CoverageMapping/logical.cpp
  clang/test/CoverageMapping/loopmacro.c
  clang/test/CoverageMapping/loops.cpp
  clang/test/CoverageMapping/macro-expansion.c
  clang/test/CoverageMapping/macro-expressions.cpp
  clang/test/CoverageMapping/macros.c
  clang/test/CoverageMapping/macroscopes.cpp
  clang/test/CoverageMapping/moremacros.c
  clang/test/CoverageMapping/return.c
  clang/test/CoverageMapping/switch.cpp
  clang/test/CoverageMapping/switchmacro.c
  clang/test/CoverageMapping/test.c
  clang/test/CoverageMapping/unreachable-macro.c
  clang/test/CoverageMapping/while.c
  clang/test/Profile/Inputs/c-general.proftext
  clang/test/Profile/branch-logical-mixed.cpp
  clang/test/Profile/c-general.c
  clang/test/Profile/cxx-lambda.cpp
  llvm/include/llvm/ProfileData/Coverage/CoverageMapping.h
  llvm/include/llvm/ProfileData/InstrProf.h
  llvm/include/llvm/ProfileData/InstrProfData.inc
  llvm/lib/ProfileData/Coverage/CoverageMapping.cpp
  llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp
  llvm/lib/ProfileData/Coverage/CoverageMappingWriter.cpp
  llvm/test/tools/llvm-cov/Inputs/binary-formats.canonical.json
  llvm/test/tools/llvm-cov/Inputs/branch-c-general.o32l
  llvm/test/tools/llvm-cov/Inputs/branch-c-general.proftext
  llvm/test/tools/llvm-cov/Inputs/branch-logical-mixed.o32l
  llvm/test/tools/llvm-cov/Inputs/branch-logical-mixed.proftext
  llvm/test/tools/llvm-cov/Inputs/branch-macros.o32l
  llvm/test/tools/llvm-cov/Inputs/branch-macros.proftext
  llvm/test/tools/llvm-cov/Inputs/branch-showBranchPercentage.o32l
  llvm/test/tools/llvm-cov/Inputs/branch-showBranchPercentage.proftext
  llvm/test/tools/llvm-cov/Inputs/branch-templates.o32l
  llvm/test/tools/llvm-cov/Inputs/branch-templates.proftext
  llvm/test/tools/llvm-cov/Inputs/showExpansions.json
  llvm/test/tools/llvm-cov/branch-c-general.c
  llvm/test/tools/llvm-cov/branch-export-json.test
  llvm/test/tools/llvm-cov/branch-export-lcov.test
  llvm/test/tools/llvm-cov/branch-logical-mixed.cpp
  llvm/test/tools/llvm-cov/branch-macros.cpp
  llvm/test/tools/llvm-cov/branch-noShowBranch.test
  llvm/test/tools/llvm-cov/branch-showBranchPercentage.c
  llvm/test/tools/llvm-cov/branch-templates.cpp
  llvm/test/tools/llvm-cov/ignore-filename-regex.test
  llvm/tools/llvm-cov/CodeCoverage.cpp
  llvm/tools/llvm-cov/CoverageExporterJson.cpp
  llvm/tools/llvm-cov/CoverageExporterLcov.cpp
  llvm/tools/llvm-cov/CoverageReport.cpp
  llvm/tools/llvm-cov/CoverageSummaryInfo.cpp
  llvm/tools/llvm-cov/CoverageSummaryInfo.h
  llvm/tools/llvm-cov/CoverageViewOptions.h
  llvm/tools/llvm-cov/SourceCoverageView.cpp
  llvm/tools/llvm-cov/SourceCoverageView.h
  llvm/tools/llvm-cov/SourceCoverageViewHTML.cpp
  llvm/tools/llvm-cov/SourceCoverageViewHTML.h
  llvm/tools/llvm-cov/SourceCoverageViewText.cpp
  llvm/tools/llvm-cov/SourceCoverageViewText.h

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D84467.280221.patch
Type: text/x-patch
Size: 205229 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20200723/fbea26bc/attachment-0001.bin>