[libc-commits] [libc] [libc][docs] Add developer guide for source-level code coverage (PR #214692)
Tapiwa Gonga via libc-commits
libc-commits at lists.llvm.org
Tue Aug 11 08:57:39 PDT 2026
https://github.com/tapiwagonga updated https://github.com/llvm/llvm-project/pull/214692
>From 9a9aab8072341dcc4146a0770d9923c5ca8cd6fd Mon Sep 17 00:00:00 2001
From: Tapiwa Gonga <tapiwagonga at google.com>
Date: Thu, 6 Aug 2026 14:22:06 +0000
Subject: [PATCH 1/2] [libc][docs] Add developer guide for source-level code
coverage
---
libc/docs/dev/code_coverage.md | 110 +++++++++++++++++++++++++++++++++
libc/docs/dev/index.md | 1 +
2 files changed, 111 insertions(+)
create mode 100644 libc/docs/dev/code_coverage.md
diff --git a/libc/docs/dev/code_coverage.md b/libc/docs/dev/code_coverage.md
new file mode 100644
index 0000000000000..9d96d5634fc53
--- /dev/null
+++ b/libc/docs/dev/code_coverage.md
@@ -0,0 +1,110 @@
+(code_coverage)=
+
+# Source-Level Code Coverage
+
+LLVM-libc supports source-level code coverage for its unit tests.
+
+Because `llvm-libc` unit tests are built as freestanding binaries (`-nostdlib`), standard `compiler-rt` coverage workflows fail in this environment as they inherently rely on the host's standard C library (`fopen`, `fwrite`), which is intentionally omitted to prevent host contamination.
+
+To bypass this constraint, the `LibcTestMain.cpp` test harness implements a custom profiling dumper:
+
+1. It silences the default `compiler-rt` dumper by overriding the global symbol: `extern "C" char __llvm_profile_filename[] = "/dev/null";`.
+2. It hooks into `atexit()` to dump coverage before the process ends. *(Note: Death tests that terminate via `_exit()`, `abort()`, or unhandled signals inherently bypass this hook. The parent test runner still correctly dumps its overall profile).*
+3. It determines the required buffer size via `__llvm_profile_get_size_for_buffer()` and allocates memory using `LIBC_NAMESPACE::linux_syscalls::mmap`.
+4. It extracts the raw profiling data from the compiler into the memory segment using `__llvm_profile_write_buffer()`.
+5. It writes the segment to a `.profraw` file using internal Linux syscall wrappers (`open`, `write`, `close`, `munmap`).
+
+## Limitations
+
+- **OS Support:** Because this relies on Linux system call wrappers, coverage extraction is strictly gated behind `#if defined(__linux__)`. On macOS or Windows builds, the coverage dumping step is gracefully bypassed, allowing the tests to compile normally.
+
+## 1. Setup and Configuration
+
+Before running any tests, you must clear old profile data and configure your CMake build directory to generate coverage instrumentation.
+
+```bash
+# 1. Clear previous profile artifacts
+find . -name "libc_cov_*.profraw" -delete
+rm -f libc_full.profdata profraw_list.txt
+
+# 2. Configure the build directory
+cmake -G Ninja -S runtimes -B build-cov \
+ -DLLVM_ENABLE_RUNTIMES="libc" \
+ -DLLVM_LIBC_FULL_BUILD=ON \
+ -DLLVM_LIBC_ENABLE_COVERAGE=ON \
+ -DCMAKE_CXX_COMPILER=clang++-19 \
+ -DCMAKE_C_COMPILER=clang-19
+```
+
+## 2. Build and Run Tests
+
+You have three options depending on how much of the library you want to test. (Targeted testing is significantly faster for local debugging).
+
+Test targets in `llvm-libc` follow a strict naming convention based on their directory structure: `libc.test.<directory_path>.<test_name>.__unit__`.
+
+### Option A: Whole Codebase
+To run the entire unit test suite (approx. 1,776 tests) and generate a massive, whole-codebase report:
+
+```bash
+ninja -C build-cov check-libc
+
+# If using Option A, do not define FILES_TO_REPORT in Step 3.
+```
+
+### Option B: Single Target
+To instantly run coverage on a single test, specify its exact Ninja target. For example, to test `isalpha`:
+
+```bash
+# Run only the specific target
+ninja -C build-cov libc.test.src.ctype.isalpha_test.__unit__
+
+# Define the source file to filter the report in Step 3
+FILES_TO_REPORT="libc/src/ctype/isalpha.cpp"
+```
+
+### Option C: Multiple Targets
+To run multiple isolated tests simultaneously, pass them as a space-separated list. For example, testing `isalpha` and `isdigit`:
+
+```bash
+# Define your targets
+TARGETS="libc.test.src.ctype.isalpha_test.__unit__ libc.test.src.ctype.isdigit_test.__unit__"
+
+# Run the targets
+ninja -C build-cov $TARGETS
+
+# Define the source files to filter the report in Step 3
+FILES_TO_REPORT="libc/src/ctype/isalpha.cpp libc/src/ctype/isdigit.cpp"
+```
+
+## 3. Generate the Report
+
+Once your tests have finished running, merge the raw profile data and extract the executables to map the coverage back to the source code.
+
+```bash
+# 1. Merge raw profiles
+find . -name "libc_cov_*.profraw" > profraw_list.txt
+llvm-profdata-19 merge -sparse --input-files=profraw_list.txt -o libc_full.profdata
+
+# 2. Extract executables
+EXECUTABLES=($(find build-cov -type f -executable -name "*__build__"))
+OBJECTS=("${EXECUTABLES[@]:1}")
+OBJECTS=("${OBJECTS[@]/#/-object=}")
+```
+
+### Choose Your Output Type
+
+You can specify the format of your final coverage report by changing the output command.
+
+**Option A: Terminal Summary Table (Text)**
+This provides a quick text-based summary of your coverage percentages directly in the terminal:
+```bash
+llvm-cov-19 report -instr-profile=libc_full.profdata "${EXECUTABLES[0]}" "${OBJECTS[@]}" $FILES_TO_REPORT
+```
+
+**Option B: Line-by-Line Interactive Webpage (HTML)**
+This generates an interactive HTML website so you can visually inspect exactly which lines of code are missing coverage. You can change the `OUTPUT_DIR` variable to save it wherever you prefer:
+```bash
+OUTPUT_DIR="coverage_html"
+llvm-cov-19 show -instr-profile=libc_full.profdata -format=html -output-dir=$OUTPUT_DIR "${EXECUTABLES[0]}" "${OBJECTS[@]}" $FILES_TO_REPORT
+```
+*(After running this, open `coverage_html/index.html` in your web browser).*
diff --git a/libc/docs/dev/index.md b/libc/docs/dev/index.md
index 4ce78540c8fad..be0d0018df249 100644
--- a/libc/docs/dev/index.md
+++ b/libc/docs/dev/index.md
@@ -8,6 +8,7 @@ Navigate to the links below for information on the respective topics:
:maxdepth: 1
building_docs
+code_coverage
code_style
source_tree_layout
entrypoints
>From d15a5d531df8783919e9966b54ff5be0681f079b Mon Sep 17 00:00:00 2001
From: Tapiwa Gonga <tapiwagonga at google.com>
Date: Tue, 11 Aug 2026 15:57:20 +0000
Subject: [PATCH 2/2] [libc][docs] Update code coverage guide with MC/DC and
freestanding execution instructions
---
libc/docs/dev/code_coverage.md | 229 +++++++++++++++++++++++++--------
1 file changed, 172 insertions(+), 57 deletions(-)
diff --git a/libc/docs/dev/code_coverage.md b/libc/docs/dev/code_coverage.md
index 9d96d5634fc53..e160cabb791c9 100644
--- a/libc/docs/dev/code_coverage.md
+++ b/libc/docs/dev/code_coverage.md
@@ -1,110 +1,225 @@
(code_coverage)=
-# Source-Level Code Coverage
+# How to Run Code Coverage and MC/DC Locally
-LLVM-libc supports source-level code coverage for its unit tests.
+LLVM-libc provides native support for generating statement, branch, and Modified Condition / Decision Coverage (MC/DC) reports locally. Because LLVM-libc runs in a freestanding environment without linking against a host standard library, coverage profile counters and boolean bitmasks are captured directly through Linux kernel system calls.
-Because `llvm-libc` unit tests are built as freestanding binaries (`-nostdlib`), standard `compiler-rt` coverage workflows fail in this environment as they inherently rely on the host's standard C library (`fopen`, `fwrite`), which is intentionally omitted to prevent host contamination.
+---
-To bypass this constraint, the `LibcTestMain.cpp` test harness implements a custom profiling dumper:
+## Prerequisites
-1. It silences the default `compiler-rt` dumper by overriding the global symbol: `extern "C" char __llvm_profile_filename[] = "/dev/null";`.
-2. It hooks into `atexit()` to dump coverage before the process ends. *(Note: Death tests that terminate via `_exit()`, `abort()`, or unhandled signals inherently bypass this hook. The parent test runner still correctly dumps its overall profile).*
-3. It determines the required buffer size via `__llvm_profile_get_size_for_buffer()` and allocates memory using `LIBC_NAMESPACE::linux_syscalls::mmap`.
-4. It extracts the raw profiling data from the compiler into the memory segment using `__llvm_profile_write_buffer()`.
-5. It writes the segment to a `.profraw` file using internal Linux syscall wrappers (`open`, `write`, `close`, `munmap`).
+* **Compiler:** Clang 18 or newer (Clang 21+ recommended for MC/DC).
+* **LLVM Profiling Tools:** `llvm-profdata` and `llvm-cov` matching the Clang version.
+* **Build System:** CMake 3.28+ and Ninja.
-## Limitations
+---
-- **OS Support:** Because this relies on Linux system call wrappers, coverage extraction is strictly gated behind `#if defined(__linux__)`. On macOS or Windows builds, the coverage dumping step is gracefully bypassed, allowing the tests to compile normally.
+## CMake Configuration
-## 1. Setup and Configuration
+Configure CMake as a standalone runtime build via `-S runtimes`.
-Before running any tests, you must clear old profile data and configure your CMake build directory to generate coverage instrumentation.
+### 1. Standard Statement & Branch Coverage
```bash
-# 1. Clear previous profile artifacts
-find . -name "libc_cov_*.profraw" -delete
-rm -f libc_full.profdata profraw_list.txt
+cmake -G Ninja -S runtimes -B build-cov \
+ -DCMAKE_C_COMPILER=clang \
+ -DCMAKE_CXX_COMPILER=clang++ \
+ -DCMAKE_BUILD_TYPE=Debug \
+ -DLLVM_ENABLE_RUNTIMES=libc \
+ -DLLVM_LIBC_FULL_BUILD=ON \
+ -DLLVM_LIBC_ENABLE_COVERAGE=ON \
+ -DLIBC_TEST_UNIT_TEST_ONLY=ON \
+ -DLIBC_TEST_SKIP_DEATH_TESTS=ON \
+ -DLIBC_TEST_SKIP_SHARED_TESTS=ON
+```
+
+### 2. MC/DC Coverage (Modified Condition / Decision Coverage)
+
+To enable MC/DC tracking in addition to statement and branch coverage, add `-DLIBC_ENABLE_MCDC=ON`:
-# 2. Configure the build directory
+```bash
cmake -G Ninja -S runtimes -B build-cov \
- -DLLVM_ENABLE_RUNTIMES="libc" \
+ -DCMAKE_C_COMPILER=clang \
+ -DCMAKE_CXX_COMPILER=clang++ \
+ -DCMAKE_BUILD_TYPE=Debug \
+ -DLLVM_ENABLE_RUNTIMES=libc \
-DLLVM_LIBC_FULL_BUILD=ON \
-DLLVM_LIBC_ENABLE_COVERAGE=ON \
- -DCMAKE_CXX_COMPILER=clang++-19 \
- -DCMAKE_C_COMPILER=clang-19
+ -DLIBC_ENABLE_MCDC=ON \
+ -DLIBC_TEST_UNIT_TEST_ONLY=ON \
+ -DLIBC_TEST_SKIP_DEATH_TESTS=ON \
+ -DLIBC_TEST_SKIP_SHARED_TESTS=ON
```
-## 2. Build and Run Tests
+---
+
+## Running Coverage Locally on a Single Function / File
+
+To measure coverage on a specific function (e.g., `isalnum` or `memchr`):
-You have three options depending on how much of the library you want to test. (Targeted testing is significantly faster for local debugging).
+### 1. Build the Targeted Unit Test
-Test targets in `llvm-libc` follow a strict naming convention based on their directory structure: `libc.test.<directory_path>.<test_name>.__unit__`.
+```bash
+ninja -C build-cov libc.test.src.ctype.isalnum_test.__unit__
+```
-### Option A: Whole Codebase
-To run the entire unit test suite (approx. 1,776 tests) and generate a massive, whole-codebase report:
+### 2. Execute the Test Binary with Profile Redirection
```bash
-ninja -C build-cov check-libc
+LLVM_PROFILE_FILE="build-cov/libc_%p.profraw" \
+ ./build-cov/libc/test/src/ctype/libc.test.src.ctype.isalnum_test.__unit__.__build__
+```
+
+### 3. Merge the Raw Profile
-# If using Option A, do not define FILES_TO_REPORT in Step 3.
+```bash
+llvm-profdata merge -sparse build-cov/libc_*.profraw -o build-cov/libc_test.profdata
```
-### Option B: Single Target
-To instantly run coverage on a single test, specify its exact Ninja target. For example, to test `isalpha`:
+### 4. View Coverage Reports in the Terminal
+**For MC/DC Truth Tables & Branch Counts:**
```bash
-# Run only the specific target
-ninja -C build-cov libc.test.src.ctype.isalpha_test.__unit__
+llvm-cov show ./build-cov/libc/test/src/ctype/libc.test.src.ctype.isalnum_test.__unit__.__build__ \
+ -instr-profile=build-cov/libc_test.profdata \
+ --show-mcdc \
+ --show-branches=count \
+ libc/src/ctype/isalnum.cpp
+```
-# Define the source file to filter the report in Step 3
-FILES_TO_REPORT="libc/src/ctype/isalpha.cpp"
+**For Standard Statement & Branch Coverage:**
+```bash
+llvm-cov show ./build-cov/libc/test/src/ctype/libc.test.src.ctype.isalnum_test.__unit__.__build__ \
+ -instr-profile=build-cov/libc_test.profdata \
+ --show-branches=count \
+ libc/src/ctype/isalnum.cpp
```
-### Option C: Multiple Targets
-To run multiple isolated tests simultaneously, pass them as a space-separated list. For example, testing `isalpha` and `isdigit`:
+Example MC/DC output:
+
+```
+ 18| 517|LLVM_LIBC_FUNCTION(int, isalnum, (int c)) {
+ 19| 517| if (c < 0 || c > cpp::numeric_limits<unsigned char>::max())
+ ------------------
+ | Branch (19:7): [True: 256, False: 261]
+ | Branch (19:16): [True: 0, False: 261]
+ ------------------
+ |---> MC/DC Decision Region (19:7) to (19:61)
+ |
+ | Number of Conditions: 2
+ | Condition C1 --> (19:7) [c < 0]
+ | Condition C2 --> (19:16) [c > max]
+ |
+ | Executed MC/DC Test Vectors:
+ | C1, C2 Result
+ | 1 { F, F = F }
+ | 2 { T, - = T }
+ |
+ | C1-Pair: covered: (1,2)
+ | C2-Pair: not covered
+ | MC/DC Coverage for Decision: 50.00%
+ ------------------
+ 20| 256| return 0;
+ 21| 261| return static_cast<int>(internal::isalnum(static_cast<char>(c)));
+ 22| 517|}
+```
+
+### 5. Run the In-Tree Patch Coverage Analyzer
+
+To generate a Markdown patch coverage summary against your current git diff:
```bash
-# Define your targets
-TARGETS="libc.test.src.ctype.isalpha_test.__unit__ libc.test.src.ctype.isdigit_test.__unit__"
+# 1. Export coverage data to JSON
+llvm-cov export ./build-cov/libc/test/src/ctype/libc.test.src.ctype.isalnum_test.__unit__.__build__ \
+ -instr-profile=build-cov/libc_test.profdata > build-cov/coverage.json
-# Run the targets
-ninja -C build-cov $TARGETS
+# 2. Generate unified diff against base commit
+git diff HEAD~1 HEAD > build-cov/patch.diff
-# Define the source files to filter the report in Step 3
-FILES_TO_REPORT="libc/src/ctype/isalpha.cpp libc/src/ctype/isdigit.cpp"
+# 3. Run the patch analyzer
+python3 libc/utils/coverage/patch_report.py build-cov/patch.diff build-cov/coverage.json
```
-## 3. Generate the Report
+---
+
+## Running Full Codebase Coverage Locally
-Once your tests have finished running, merge the raw profile data and extract the executables to map the coverage back to the source code.
+To build and measure coverage across all unit tests in the entire LLVM-libc codebase:
+
+### 1. Clean Previous Profile Counters
+
+```bash
+rm -f build-cov/libc_cov_*.profraw build-cov/libc_full.profdata
+```
+
+### 2. Execute All Unit Tests Across the Codebase
+
+The `-k 0` flag ensures Ninja executes all targets across all subsystems even if an isolated test fails:
```bash
-# 1. Merge raw profiles
-find . -name "libc_cov_*.profraw" > profraw_list.txt
-llvm-profdata-19 merge -sparse --input-files=profraw_list.txt -o libc_full.profdata
+export LLVM_PROFILE_FILE="build-cov/libc_cov_%p.profraw"
+ninja -k 0 -C build-cov libc-unit-tests || true
+```
+
+### 3. Merge All Collected Raw Profiles
+
+```bash
+find build-cov -name "libc_cov_*.profraw" > build-cov/profraw_list.txt
+llvm-profdata merge -sparse --input-files=build-cov/profraw_list.txt -o build-cov/libc_full.profdata
+```
-# 2. Extract executables
+### 4. Collect Test Executables and Export Coverage JSON
+
+```bash
+# Gather all test binaries
EXECUTABLES=($(find build-cov -type f -executable -name "*__build__"))
OBJECTS=("${EXECUTABLES[@]:1}")
OBJECTS=("${OBJECTS[@]/#/-object=}")
+
+# Export JSON data
+llvm-cov export \
+ -format=text \
+ -instr-profile=build-cov/libc_full.profdata \
+ "${EXECUTABLES[0]}" "${OBJECTS[@]}" \
+ -ignore-filename-regex=".*(test|utils).*" > build-cov/coverage.json
+```
+
+### 5. Generate Interactive HTML Report
+
+**With MC/DC Truth Tables:**
+```bash
+llvm-cov show \
+ -format=html \
+ -output-dir=coverage_html \
+ -instr-profile=build-cov/libc_full.profdata \
+ "${EXECUTABLES[0]}" "${OBJECTS[@]}" \
+ --show-directory-coverage \
+ --show-branches=count \
+ --show-mcdc \
+ --show-mcdc-summary \
+ -ignore-filename-regex=".*(test|utils).*"
```
-### Choose Your Output Type
+**For Standard Statement Coverage (without MC/DC):**
+```bash
+llvm-cov show \
+ -format=html \
+ -output-dir=coverage_html \
+ -instr-profile=build-cov/libc_full.profdata \
+ "${EXECUTABLES[0]}" "${OBJECTS[@]}" \
+ --show-directory-coverage \
+ --show-branches=count \
+ -ignore-filename-regex=".*(test|utils).*"
+```
-You can specify the format of your final coverage report by changing the output command.
+### 6. Print the Full Codebase Coverage Summary
-**Option A: Terminal Summary Table (Text)**
-This provides a quick text-based summary of your coverage percentages directly in the terminal:
```bash
-llvm-cov-19 report -instr-profile=libc_full.profdata "${EXECUTABLES[0]}" "${OBJECTS[@]}" $FILES_TO_REPORT
+python3 libc/utils/coverage/full_report.py build-cov/coverage.json
```
-**Option B: Line-by-Line Interactive Webpage (HTML)**
-This generates an interactive HTML website so you can visually inspect exactly which lines of code are missing coverage. You can change the `OUTPUT_DIR` variable to save it wherever you prefer:
+### 7. View the HTML Dashboard in Your Browser
+
```bash
-OUTPUT_DIR="coverage_html"
-llvm-cov-19 show -instr-profile=libc_full.profdata -format=html -output-dir=$OUTPUT_DIR "${EXECUTABLES[0]}" "${OBJECTS[@]}" $FILES_TO_REPORT
+xdg-open coverage_html/index.html
```
-*(After running this, open `coverage_html/index.html` in your web browser).*
More information about the libc-commits
mailing list