[libc-commits] [libc] [libc][docs] Add developer guide for source-level code coverage (PR #214692)

Thu Aug 20 03:53:47 PDT 2026

https://github.com/tapiwagonga updated https://github.com/llvm/llvm-project/pull/214692

>From 6999b626021b3e0fece5798ebfd6d90dd2b1fa22 Mon Sep 17 00:00:00 2001
From: Tapiwa Gonga <tapiwagonga at google.com>
Date: Wed, 19 Aug 2026 14:28:48 +0000
Subject: [PATCH] [libc][docs] Add developer guide for code coverage and MC/DC

---
 libc/docs/dev/code_coverage.md | 316 +++++++++++++++++++++++++++++++++
 libc/docs/dev/index.md         |   1 +
 2 files changed, 317 insertions(+)
 create mode 100644 libc/docs/dev/code_coverage.md

diff --git a/libc/docs/dev/code_coverage.md b/libc/docs/dev/code_coverage.md
new file mode 100644
index 0000000000000..de01a71bbcf79
--- /dev/null
+++ b/libc/docs/dev/code_coverage.md
@@ -0,0 +1,316 @@
+# Code Coverage in LLVM-libc
+
+(code_coverage)=
+
+This document describes how to configure, generate, and view code coverage and Modified Condition / Decision Coverage (MC/DC) reports for LLVM-libc.
+
+---
+
+## Overview
+
+Code coverage measures the proportion of source code executed during testing. In LLVM-libc, coverage metrics identify untested edge cases, prevent regressions across supported architectures, and provide verification evidence for safety-critical systems.
+
+### Coverage Modes
+
+* **Statement and Branch Coverage:**  
+  Measures line execution and verifies whether conditional branches evaluated to both true and false paths.
+
+* **Modified Condition / Decision Coverage (MC/DC):**  
+  Evaluates boolean conditions within compound decisions (such as `if (A && B)`). Verifies that each individual sub-condition is tested with both true and false values and independently affects the outcome of the enclosing decision. Required by safety-critical standards such as DO-178C (aviation) and ISO 26262 (automotive).
+
+---
+
+## Prerequisites
+
+Generating coverage reports requires Clang, LLVM profile tools, CMake, and Ninja:
+
+* **Compiler:** Clang 18 or later (Clang 21 or later is required for MC/DC instrumentation).
+* **LLVM Utilities:** Matching versions of `llvm-profdata` and `llvm-cov`.
+* **Build System:** CMake 3.28+ and Ninja.
+
+### Toolchain Installation & Setup (Debian/Ubuntu)
+
+Install the required packages and configure the default tool links:
+
+```bash
+sudo apt install clang llvm lld ninja-build cmake
+
+# If using versioned LLVM packages (e.g. LLVM 21), link the unversioned utilities:
+sudo update-alternatives --install /usr/bin/llvm-profdata llvm-profdata /usr/bin/llvm-profdata-21 100
+sudo update-alternatives --install /usr/bin/llvm-cov llvm-cov /usr/bin/llvm-cov-21 100
+```
+
+:::{note}
+Compiling the full suite of unit tests with coverage instrumentation encompasses ~10,000 target nodes. On multi-core workstations, parallel compilation completes in a few minutes. On resource-constrained systems, virtual machines with 2–4 cores, or cold builds without compiler caching, the initial build may take up to 30 minutes.
+:::
+
+---
+
+## Cleaning Profile Counters
+
+Before running a new coverage test pass, remove existing `.profraw` and `.profdata` files to avoid merging stale profiling data:
+
+```bash
+find build-cov -name "libc_cov_*.profraw" -delete 2>/dev/null || true
+rm -f build-cov/libc_full.profdata
+```
+
+---
+
+## How to Run Standard Coverage (Statement and Branch)
+
+Standard coverage records line execution and branch direction metrics across all LLVM-libc entrypoints and support routines.
+
+### 1. CMake Configuration
+
+Configure LLVM-libc with `-DLLVM_LIBC_ENABLE_COVERAGE=ON`:
+
+```bash
+cmake -G Ninja -S runtimes -B build-cov \
+  -DCMAKE_C_COMPILER=clang \
+  -DCMAKE_CXX_COMPILER=clang++ \
+  -DCMAKE_BUILD_TYPE=Debug \
+  -DLLVM_ENABLE_RUNTIMES=libc \
+  -DLLVM_LIBC_FULL_BUILD=ON \
+  -DLLVM_LIBC_ENABLE_COVERAGE=ON \
+  -DLIBC_ENABLE_MCDC=OFF \
+  -DLIBC_TEST_UNIT_TEST_ONLY=ON \
+  -DLIBC_TEST_SKIP_DEATH_TESTS=ON
+```
+
+### 2. Build Unit Tests
+
+Compile the test binaries:
+
+```bash
+ninja -k 0 -C build-cov libc-unit-tests || true
+```
+
+### 3. Run Unit Tests
+
+Execute the compiled unit test binaries in parallel across available CPU cores:
+
+```bash
+# Clean stale counters
+find build-cov -name "libc_cov_*.profraw" -delete 2>/dev/null || true
+rm -f build-cov/libc_full.profdata
+
+# Run all test binaries with isolated PID profile names
+export LLVM_PROFILE_FILE="libc_cov_%p.profraw"
+(cd build-cov && find libc/test -type f -executable -name "*__build__" | xargs -P $(nproc) -I {} sh -c '{} > /dev/null 2>&1 || true')
+```
+
+### 4. Merge Profile Counters
+
+Merge all emitted raw profile counters into an indexed profile dataset:
+
+```bash
+find build-cov -name "libc_cov_*.profraw" > profraw_list.txt
+llvm-profdata merge -sparse --input-files=profraw_list.txt -o build-cov/libc_full.profdata
+```
+
+### 5. View Coverage Reports
+
+Collect test binary object references:
+
+```bash
+TEST_BINARIES=$(find build-cov/libc/test -type f -executable -name "*__build__" | sed 's/^/-object=/')
+FIRST_BIN=$(echo "$TEST_BINARIES" | head -n 1 | sed 's/^-object=//')
+OTHER_BINS=$(echo "$TEST_BINARIES" | tail -n +2)
+```
+
+#### Terminal Summary Table
+
+Display a directory-by-directory coverage report in the terminal:
+
+```bash
+llvm-cov report \
+  -instr-profile=build-cov/libc_full.profdata \
+  "$FIRST_BIN" $OTHER_BINS \
+  --show-branch-summary \
+  -ignore-filename-regex=".*(test|utils).*"
+```
+
+#### Interactive HTML Dashboard
+
+Generate a browsable HTML site with source file coverage drill-downs:
+
+```bash
+llvm-cov show \
+  -format=html \
+  -output-dir=coverage_html \
+  -instr-profile=build-cov/libc_full.profdata \
+  "$FIRST_BIN" $OTHER_BINS \
+  --show-directory-coverage \
+  --show-branches=count \
+  -ignore-filename-regex=".*(test|utils).*"
+
+# Open in browser
+xdg-open coverage_html/index.html
+```
+
+---
+
+## How to Run Modified Condition / Decision Coverage (MC/DC)
+
+MC/DC instrumentation captures condition-level truth tables for compound logical expressions in addition to statement and branch metrics.
+
+### 1. CMake Configuration
+
+Enable MC/DC instrumentation by adding `-DLIBC_ENABLE_MCDC=ON`:
+
+```bash
+cmake -G Ninja -S runtimes -B build-cov \
+  -DCMAKE_C_COMPILER=clang \
+  -DCMAKE_CXX_COMPILER=clang++ \
+  -DCMAKE_BUILD_TYPE=Debug \
+  -DLLVM_ENABLE_RUNTIMES=libc \
+  -DLLVM_LIBC_FULL_BUILD=ON \
+  -DLLVM_LIBC_ENABLE_COVERAGE=ON \
+  -DLIBC_ENABLE_MCDC=ON \
+  -DLIBC_TEST_UNIT_TEST_ONLY=ON \
+  -DLIBC_TEST_SKIP_DEATH_TESTS=ON
+```
+
+### 2. Build Unit Tests
+
+Compile the test binaries:
+
+```bash
+ninja -k 0 -C build-cov libc-unit-tests || true
+```
+
+### 3. Run Unit Tests
+
+Execute the compiled unit test binaries in parallel:
+
+```bash
+# Clean stale counters
+find build-cov -name "libc_cov_*.profraw" -delete 2>/dev/null || true
+rm -f build-cov/libc_full.profdata
+
+# Run all test binaries with isolated PID profile names
+export LLVM_PROFILE_FILE="libc_cov_%p.profraw"
+(cd build-cov && find libc/test -type f -executable -name "*__build__" | xargs -P $(nproc) -I {} sh -c '{} > /dev/null 2>&1 || true')
+```
+
+### 4. Merge Profile Counters
+
+Merge the raw counters into an indexed profile dataset:
+
+```bash
+find build-cov -name "libc_cov_*.profraw" > profraw_list.txt
+llvm-profdata merge -sparse --input-files=profraw_list.txt -o build-cov/libc_full.profdata
+```
+
+### 5. View MC/DC Coverage Reports
+
+Collect test binary object references:
+
+```bash
+TEST_BINARIES=$(find build-cov/libc/test -type f -executable -name "*__build__" | sed 's/^/-object=/')
+FIRST_BIN=$(echo "$TEST_BINARIES" | head -n 1 | sed 's/^-object=//')
+OTHER_BINS=$(echo "$TEST_BINARIES" | tail -n +2)
+```
+
+#### Terminal Summary Table with MC/DC Metrics
+
+Display statement, branch, and MC/DC decision coverage percentages in the terminal:
+
+```bash
+llvm-cov report \
+  -instr-profile=build-cov/libc_full.profdata \
+  "$FIRST_BIN" $OTHER_BINS \
+  --show-branch-summary \
+  --show-mcdc-summary \
+  -ignore-filename-regex=".*(test|utils).*"
+```
+
+#### Interactive HTML Dashboard with MC/DC Analysis
+
+Generate an interactive HTML dashboard containing MC/DC decision breakdown tables and line-by-line coverage:
+
+```bash
+llvm-cov show \
+  -format=html \
+  -output-dir=coverage_html \
+  -instr-profile=build-cov/libc_full.profdata \
+  "$FIRST_BIN" $OTHER_BINS \
+  --show-directory-coverage \
+  --show-branches=count \
+  --show-mcdc \
+  --show-mcdc-summary \
+  -ignore-filename-regex=".*(test|utils).*"
+
+# Open in browser
+xdg-open coverage_html/index.html
+```
+
+---
+
+## Interpreting Results
+
+Understanding coverage metrics helps developers assess test completeness, identify uncovered edge cases, and author targeted unit tests.
+
+### Terminal Summary Metrics
+
+When executing `llvm-cov report`, the terminal output summarizes coverage across files and directories:
+
+* **Region Coverage:**  
+  Measures execution of discrete Abstract Syntax Tree (AST) expression sub-blocks (such as the body of an `if` statement or ternary expressions). A lower region coverage than line coverage indicates partially executed expressions on lines that were counted as hit.
+
+* **Line Coverage:**  
+  Tracks physical source lines executed during the test run. Unexecuted lines represent functions, conditional branches, or error recovery handlers that were never invoked.
+
+* **Branch Coverage:**  
+  Evaluates conditional branch outcomes. If a branch indicates `50%` coverage, the condition was only ever evaluated in one direction (for example, always `True`), leaving the alternative path (`False`) untested.
+
+* **MC/DC Coverage:**  
+  Reports the percentage of compound boolean decisions where every atomic sub-condition was demonstrated to independently determine the final decision outcome.
+
+### HTML Dashboard and Source Inspection
+
+The interactive HTML dashboard (`coverage_html/index.html`) provides line-by-line visual inspection of source implementations:
+
+#### Line Execution Highlights
+
+* **Green Lines:** Source code executed by tests. The margin integer indicates execution count.
+* **Red Lines:** Unexecuted code that requires additional unit test coverage.
+
+#### Branch Markers
+
+Conditional statements display branch hit counts inline in the format `[True: N, False: M]`. An entry of `[True: 10, False: 0]` indicates that the conditional expression was never evaluated as False during testing. Adding a unit test where the condition evaluates to False resolves this gap.
+
+#### MC/DC Truth Tables and Condition Diagnostics
+
+When `--show-mcdc` is enabled, `llvm-cov` renders a boolean truth table directly below compound decisions:
+
+```
+   19|    517|  if (c < 0 || c > 255)
+  ------------------
+  |  Executed MC/DC Test Vectors:
+  |     C1, C2    Result
+  |  1 { F,  F  = F      }
+  |  2 { T,  -  = T      }
+  |
+  |  C1-Pair: covered (1, 2)
+  |  C2-Pair: not covered
+  |  MC/DC Coverage for Decision: 50.00%
+  ------------------
+```
+
+##### Understanding the Truth Table
+
+1. **Identify the Conditions:**  
+   In `if (c < 0 || c > 255)`, condition **C1** is `c < 0` and **C2** is `c > 255`.
+
+2. **Inspect Executed Vectors:**  
+   * **Vector 1 (`F, F = F`):** Tested with an in-range value (e.g. `c = 100`). Both C1 and C2 evaluated to False, producing a False outcome.
+   * **Vector 2 (`T, - = T`):** Tested with a negative value (e.g. `c = -1`). C1 evaluated to True, which immediately satisfied the `if` statement (C2 was short-circuited `-`).
+
+3. **Evaluate Coverage Status:**  
+   * **`C1-Pair: covered (1, 2)`:** Verified. Comparing Vector 1 and Vector 2 proves that toggling C1 alone flips the overall decision outcome.
+   * **`C2-Pair: not covered`:** Missing. C2 was never tested in a state where it independently caused the `if` condition to become True.
+
+4. **How to Fix the Gap:**  
+   Add a unit test case with `c = 256`. This evaluates C1 as False and C2 as True (`3 { F, T = T }`), forming the missing independence pair `(1, 3)` for C2 and achieving 100% MC/DC coverage.
diff --git a/libc/docs/dev/index.md b/libc/docs/dev/index.md
index 4ce78540c8fad..be0d0018df249 100644
--- a/libc/docs/dev/index.md
+++ b/libc/docs/dev/index.md
@@ -8,6 +8,7 @@ Navigate to the links below for information on the respective topics:
 :maxdepth: 1
 
 building_docs
+code_coverage
 code_style
 source_tree_layout
 entrypoints

