[libc-commits] [libc] [llvm] [libc][ci] Add automated pre-commit and post-commit coverage bots (PR #215269)
Tapiwa Gonga via libc-commits
libc-commits at lists.llvm.org
Mon Aug 10 05:56:10 PDT 2026
https://github.com/tapiwagonga created https://github.com/llvm/llvm-project/pull/215269
This patch adds automated GitHub Actions workflows and supporting Python tools to generate code coverage reports for LLVM-libc.
**Pre-commit Patch Coverage Bot (.github/workflows/libc-pre-commit-coverage.yml):**
When a pull request modifies C/C++ files under libc/src/, libc/include/, or libc/test/, the workflow parses the modified file list, maps each source file to its corresponding Ninja unit test target, and executes only those specific tests. The workflow then diffs the patch against the base commit, extracts statement execution data from llvm-cov export, and posts a line-level coverage report to the job summary and PR comments. Pull requests modifying only documentation, benchmarks, or Markdown files are ignored.
**Post-commit Global Coverage Bot (.github/workflows/libc-post-commit-coverage.yml):**
Runs on pushes to main and scheduled nightly builds. It executes the full LLVM-libc unit test suite (1,920+ tests), generates an interactive HTML report using llvm-cov show with directory grouping (--show-directory-coverage) and branch execution counts (--show-branches=count), and deploys the static dashboard to GitHub Pages. It creates .nojekyll to prevent 404 errors on internal directories starting with underscores (libc/src/__support/).
**Coverage Report Generation:**
libc/utils/coverage/patch_report.py compares git diff against llvm-cov JSON output to evaluate line coverage for newly added or modified lines.
libc/utils/coverage/full_report.py aggregates global line and function coverage across all LLVM-libc subsystems.
>From 08778902ac2fbe6a6f6279fe90b21c8fe5a90d93 Mon Sep 17 00:00:00 2001
From: Tapiwa Gonga <tapiwagonga at google.com>
Date: Mon, 10 Aug 2026 11:13:29 +0000
Subject: [PATCH] [libc][ci] Add automated pre-commit and post-commit coverage
bots
---
.../workflows/libc-post-commit-coverage.yml | 134 +++++++
.../workflows/libc-pre-commit-coverage.yml | 362 ++++++++++++++++++
libc/utils/coverage/full_report.py | 138 +++++++
libc/utils/coverage/patch_report.py | 245 ++++++++++++
4 files changed, 879 insertions(+)
create mode 100644 .github/workflows/libc-post-commit-coverage.yml
create mode 100644 .github/workflows/libc-pre-commit-coverage.yml
create mode 100644 libc/utils/coverage/full_report.py
create mode 100644 libc/utils/coverage/patch_report.py
diff --git a/.github/workflows/libc-post-commit-coverage.yml b/.github/workflows/libc-post-commit-coverage.yml
new file mode 100644
index 0000000000000..6a7749b5dc181
--- /dev/null
+++ b/.github/workflows/libc-post-commit-coverage.yml
@@ -0,0 +1,134 @@
+name: libc post-commit code coverage
+
+permissions:
+ contents: write # Required to publish live HTML dashboard to gh-pages branch
+
+on:
+ # Daily overnight run at 02:00 UTC
+ schedule:
+ - cron: '0 2 * * *'
+
+ # Allow manual on-demand execution from GitHub Actions UI
+ workflow_dispatch:
+
+ # Trigger on pushes to main / post-commit-bot
+ push:
+ branches:
+ - main
+ - post-commit-bot
+ paths:
+ - 'libc/**'
+ - '.github/workflows/libc-post-commit-coverage.yml'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: true
+
+jobs:
+ full-coverage:
+ timeout-minutes: 60
+ name: libc-full-coverage
+ runs-on: ubuntu-24.04
+ container:
+ image: ghcr.io/llvm/libc-ubuntu-24.04:latest at sha256:8fee4c9ce5a1fd095686593cd36e032c48a31b6ae575378c82accd1d86a08d59
+ options: --privileged
+ defaults:
+ run:
+ shell: bash
+ steps:
+ - name: Checkout Code
+ uses: actions/checkout at v4
+ with:
+ fetch-depth: 1
+ persist-credentials: false
+
+ - name: Setup Compiler Cache (sccache)
+ uses: hendrikmuhs/ccache-action at v1.2.23
+ with:
+ max-size: 1G
+ key: libc_coverage_unified_v2_x86_64
+ variant: sccache
+
+ - name: Configure CMake
+ run: |
+ echo "=== [STAGE 1/4] Starting CMake Configuration at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ START_TIME=$(date +%s)
+
+ export CMAKE_FLAGS="
+ -G Ninja
+ -S runtimes
+ -B build-cov
+ -DCMAKE_C_COMPILER=clang-23
+ -DCMAKE_CXX_COMPILER=clang++-23
+ -DCMAKE_BUILD_TYPE=Debug
+ -DCMAKE_C_COMPILER_LAUNCHER=sccache
+ -DCMAKE_CXX_COMPILER_LAUNCHER=sccache
+ -DLLVM_USE_LINKER=lld-23
+ -DLLVM_ENABLE_RUNTIMES=libc
+ -DLLVM_LIBC_FULL_BUILD=ON
+ -DLLVM_LIBC_ENABLE_COVERAGE=ON
+ -DLIBC_TEST_UNIT_TEST_ONLY=ON
+ -DLIBC_TEST_SKIP_DEATH_TESTS=ON
+ -DLIBC_TEST_SKIP_SHARED_TESTS=ON
+ "
+ cmake $CMAKE_FLAGS
+
+ END_TIME=$(date +%s)
+ echo "=== [STAGE 1/4] CMake Configuration Completed in $((END_TIME - START_TIME))s ==="
+
+ - name: Run Full Codebase Unit Tests
+ run: |
+ echo "=== [STAGE 2/4] Running Full Unit Test Suite at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ START_TIME=$(date +%s)
+
+ export LLVM_PROFILE_FILE="libc_cov_%p.profraw"
+ ninja -C build-cov libc-unit-tests
+
+ END_TIME=$(date +%s)
+ echo "=== [STAGE 2/4] libc-unit-tests Completed in $((END_TIME - START_TIME))s ==="
+
+ - name: Merge Profiles
+ run: |
+ echo "=== [STAGE 3/4] Merging Raw Profiles at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ find build-cov -name "libc_cov_*.profraw" > profraw_list.txt
+ NUM_PROFS=$(wc -l < profraw_list.txt || echo 0)
+ echo "[LOG] Discovered $NUM_PROFS raw profile data files."
+ llvm-profdata-23 merge -sparse --input-files=profraw_list.txt -o libc_full.profdata
+ echo "[LOG] Merged full codebase profile into libc_full.profdata."
+
+ - name: Generate Reports and Summary
+ env:
+ COMMIT_SHA: ${{ github.sha }}
+ BRANCH_REF: ${{ github.ref_name }}
+ run: |
+ echo "=== [STAGE 4/4] Generating Full Codebase Coverage Reports at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ EXECUTABLES=($(find build-cov -type f -executable -name "*__build__"))
+ OBJECTS=("${EXECUTABLES[@]:1}")
+ OBJECTS=("${OBJECTS[@]/#/-object=}")
+
+ echo "[LOG] Exporting coverage data across ${#EXECUTABLES[@]} test binaries."
+
+ # 1. Generate JSON export for Full Coverage Analyzer
+ llvm-cov-23 export -format=text -instr-profile=libc_full.profdata "${EXECUTABLES[0]}" "${OBJECTS[@]}" > coverage.json
+
+ # 2. Generate HTML Coverage Report with directory hierarchy and branch tracking
+ llvm-cov-23 show -format=html \
+ -output-dir=coverage_html \
+ -instr-profile=libc_full.profdata \
+ "${EXECUTABLES[0]}" "${OBJECTS[@]}" \
+ --show-directory-coverage \
+ --show-branches=count \
+ -ignore-filename-regex=".*(test|utils).*"
+ touch coverage_html/.nojekyll
+
+ # 3. Run Full Coverage Analyzer
+ python3 libc/utils/coverage/full_report.py coverage.json "$COMMIT_SHA" "$BRANCH_REF" >> $GITHUB_STEP_SUMMARY
+ echo "[LOG] Post-commit summary report written to GITHUB_STEP_SUMMARY."
+
+ - name: Deploy Coverage Dashboard to GitHub Pages
+ uses: peaceiris/actions-gh-pages at v4
+ with:
+ github_token: ${{ secrets.GITHUB_TOKEN }}
+ publish_dir: ./coverage_html
+ enable_jekyll: false
+ force_orphan: true
diff --git a/.github/workflows/libc-pre-commit-coverage.yml b/.github/workflows/libc-pre-commit-coverage.yml
new file mode 100644
index 0000000000000..ae4eb224cfc21
--- /dev/null
+++ b/.github/workflows/libc-pre-commit-coverage.yml
@@ -0,0 +1,362 @@
+name: libc pre-commit code coverage
+
+permissions:
+ contents: read
+ pull-requests: write # Required to post/update pre-commit coverage comments
+
+on:
+ workflow_dispatch:
+ push:
+ branches:
+ - pre-commit-bot
+ - main
+ paths:
+ - 'libc/src/**'
+ - 'libc/include/**'
+ - 'libc/test/**'
+ - 'libc/CMakeLists.txt'
+ - '.github/workflows/libc-pre-commit-coverage.yml'
+ - '!libc/docs/**'
+ - '!libc/benchmarks/**'
+ - '!libc/fuzzing/**'
+ - '!libc/utils/**'
+ - '!**.md'
+ pull_request:
+ paths:
+ - 'libc/src/**'
+ - 'libc/include/**'
+ - 'libc/test/**'
+ - 'libc/CMakeLists.txt'
+ - '.github/workflows/libc-pre-commit-coverage.yml'
+ - '!libc/docs/**'
+ - '!libc/benchmarks/**'
+ - '!libc/fuzzing/**'
+ - '!libc/utils/**'
+ - '!**.md'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+ cancel-in-progress: true
+
+jobs:
+ pre-commit-coverage:
+ timeout-minutes: 60
+ name: libc-pre-commit-coverage
+ runs-on: ubuntu-24.04
+ container:
+ image: ghcr.io/llvm/libc-ubuntu-24.04:latest at sha256:8fee4c9ce5a1fd095686593cd36e032c48a31b6ae575378c82accd1d86a08d59
+ options: >-
+ --privileged
+ defaults:
+ run:
+ shell: bash
+ steps:
+ - name: Checkout Code
+ uses: actions/checkout at v4
+ with:
+ ref: ${{ github.event.pull_request.head.sha || github.sha }}
+ fetch-depth: 2
+ persist-credentials: false
+
+ - name: Setup Compiler Cache (sccache)
+ uses: hendrikmuhs/ccache-action at v1.2.23
+ with:
+ max-size: 1G
+ key: libc_coverage_unified_v2_x86_64
+ variant: sccache
+
+ - name: Configure CMake
+ run: |
+ echo "=== [STAGE 1/4] Starting CMake Configuration at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ START_TIME=$(date +%s)
+
+ export CMAKE_FLAGS="
+ -G Ninja
+ -S runtimes
+ -B build-cov
+ -DCMAKE_C_COMPILER=clang-23
+ -DCMAKE_CXX_COMPILER=clang++-23
+ -DCMAKE_BUILD_TYPE=Debug
+ -DCMAKE_C_COMPILER_LAUNCHER=sccache
+ -DCMAKE_CXX_COMPILER_LAUNCHER=sccache
+ -DLLVM_USE_LINKER=lld-23
+ -DLLVM_ENABLE_RUNTIMES=libc
+ -DLLVM_LIBC_FULL_BUILD=ON
+ -DLLVM_LIBC_ENABLE_COVERAGE=ON
+ -DLIBC_TEST_UNIT_TEST_ONLY=ON
+ -DLIBC_TEST_SKIP_DEATH_TESTS=ON
+ -DLIBC_TEST_SKIP_SHARED_TESTS=ON
+ "
+ cmake $CMAKE_FLAGS
+
+ END_TIME=$(date +%s)
+ echo "=== [STAGE 1/4] CMake Configuration Completed in $((END_TIME - START_TIME))s ==="
+
+ - name: Build and Run Targeted Tests
+ env:
+ EVENT_NAME: ${{ github.event_name }}
+ PR_BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+ PR_BASE_REF: ${{ github.event.pull_request.base.ref }}
+ PR_HEAD_REF: ${{ github.event.pull_request.head.ref }}
+ PR_BASE_REPO: ${{ github.event.pull_request.base.repo.full_name }}
+ PR_HEAD_REPO: ${{ github.event.pull_request.head.repo.full_name }}
+ COMMIT_SHA: ${{ github.sha }}
+ BRANCH_REF: ${{ github.ref_name }}
+ REPO_NAME: ${{ github.repository }}
+ run: |
+ echo "=== [STAGE 2/4] Starting Targeted Test Discovery at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ START_TIME=$(date +%s)
+
+ # Ensure upstream llvm-project remote is configured
+ git remote add upstream https://github.com/llvm/llvm-project.git 2>/dev/null || true
+
+ # 1. Deterministically resolve Base and Head references against upstream llvm-project
+ if [ "$EVENT_NAME" == "pull_request" ]; then
+ BASE_SHA="$PR_BASE_SHA"
+ HEAD_SHA="$PR_HEAD_SHA"
+ BASE_REF="$PR_BASE_REF"
+ HEAD_REF="$PR_HEAD_REF"
+ BASE_REPO="${PR_BASE_REPO:-llvm/llvm-project}"
+ HEAD_REPO="${PR_HEAD_REPO:-$REPO_NAME}"
+ git fetch upstream "$BASE_SHA" --depth=1 2>/dev/null || git fetch origin "$BASE_SHA" --depth=1 2>/dev/null || git fetch origin "$BASE_REF" --depth=1 2>/dev/null || true
+ DIFF_BASE="$BASE_SHA"
+ else
+ # For direct pushes, compare the pushed commit against its parent (HEAD~1)
+ BASE_SHA=$(git rev-parse HEAD~1 2>/dev/null || echo "HEAD~1")
+ HEAD_SHA="$COMMIT_SHA"
+ BASE_REF="main"
+ HEAD_REF="$BRANCH_REF"
+ BASE_REPO="llvm/llvm-project"
+ HEAD_REPO="$REPO_NAME"
+ DIFF_BASE="HEAD~1"
+ fi
+
+ # Persist resolved commit references for all downstream steps
+ echo "DIFF_BASE=$DIFF_BASE" >> $GITHUB_ENV
+ echo "BASE_SHA=$BASE_SHA" >> $GITHUB_ENV
+ echo "HEAD_SHA=$HEAD_SHA" >> $GITHUB_ENV
+ echo "BASE_REF=$BASE_REF" >> $GITHUB_ENV
+ echo "HEAD_REF=$HEAD_REF" >> $GITHUB_ENV
+ echo "BASE_REPO=$BASE_REPO" >> $GITHUB_ENV
+ echo "HEAD_REPO=$HEAD_REPO" >> $GITHUB_ENV
+
+ echo "[LOG] Resolved Base: ${BASE_REF} (${BASE_SHA:0:7}) in ${BASE_REPO}"
+ echo "[LOG] Resolved Head: ${HEAD_REF} (${HEAD_SHA:0:7}) in ${HEAD_REPO}"
+
+ MODIFIED_FILES=$(git diff --name-only "$DIFF_BASE" HEAD -- libc/src/ || true)
+ echo "[LOG] Modified files in libc/src/:"
+ echo "$MODIFIED_FILES"
+
+ if [ -z "$MODIFIED_FILES" ]; then
+ echo "[LOG] No source files modified in libc/src/. Exiting successfully."
+ echo "TARGETS=" >> $GITHUB_ENV
+
+ echo "## LLVM-libc Patch Coverage Report" > coverage_report.md
+ echo "" >> coverage_report.md
+ echo "- **Base Branch:** [\`${BASE_REF}\` (${BASE_SHA:0:7})](https://github.com/${BASE_REPO}/commit/${BASE_SHA})" >> coverage_report.md
+ echo "- **Head Commit:** [\`${HEAD_REF}\` (${HEAD_SHA:0:7})](https://github.com/${HEAD_REPO}/commit/${HEAD_SHA})" >> coverage_report.md
+ echo "---" >> coverage_report.md
+ echo "" >> coverage_report.md
+ echo "**Coverage Validated**" >> coverage_report.md
+ echo "No \`.cpp\` source files in \`libc/src/\` were modified in this patch." >> coverage_report.md
+
+ cat coverage_report.md >> $GITHUB_STEP_SUMMARY
+ exit 0
+ fi
+
+ # Get all valid ninja targets
+ ninja -C build-cov -t targets > all_targets.txt
+
+ TARGETS=""
+
+ for FILE in $MODIFIED_FILES; do
+ # Skip core support files from direct target mapping (tested transitively)
+ if [[ "$FILE" == *"__support/"* ]]; then
+ echo "[LOG] -> Support file ($FILE) will be tested transitively."
+ continue
+ fi
+
+ if [[ "$FILE" =~ ^libc/src/([^/]+)/(.*/)?([^/]+)\.cpp$ ]]; then
+ DIR="${BASH_REMATCH[1]}"
+ FUNC="${BASH_REMATCH[3]}"
+ ALL_MATCHES=$(grep -oE "^libc\.test\.src\.${DIR}\.([a-zA-Z0-9_]+\.)*${FUNC}_test\.__unit__" all_targets.txt || true)
+ FOUND_TARGET=$(echo "$ALL_MATCHES" | head -n 1 | tr -d '\r\n ' || true)
+ if [ -n "$FOUND_TARGET" ]; then
+ echo "[LOG] -> Matched $FILE -> $FOUND_TARGET"
+ TARGETS="$TARGETS $FOUND_TARGET"
+ else
+ echo "[LOG] -> Notice: No direct unit test target found for $FILE."
+ fi
+ fi
+ done
+
+ # Remove duplicate targets
+ TARGETS=$(echo "$TARGETS" | xargs -n1 | sort -u | xargs || true)
+ echo "[LOG] Executing Ninja targets: $TARGETS"
+
+ if [ -n "$TARGETS" ]; then
+ export LLVM_PROFILE_FILE="libc_cov_%p.profraw"
+ ninja -C build-cov $TARGETS
+ echo "TARGETS=$TARGETS" >> $GITHUB_ENV
+ else
+ echo "[LOG] No standalone unit test targets to run."
+ echo "TARGETS=" >> $GITHUB_ENV
+
+ echo "## LLVM-libc Patch Coverage Report" > coverage_report.md
+ echo "" >> coverage_report.md
+ echo "- **Base Branch:** [\`${BASE_REF}\` (${BASE_SHA:0:7})](https://github.com/${BASE_REPO}/commit/${BASE_SHA})" >> coverage_report.md
+ echo "- **Head Commit:** [\`${HEAD_REF}\` (${HEAD_SHA:0:7})](https://github.com/${HEAD_REPO}/commit/${HEAD_SHA})" >> coverage_report.md
+ echo "---" >> coverage_report.md
+ echo "" >> coverage_report.md
+ echo "**Coverage Validated**" >> coverage_report.md
+ echo "Modified files do not have standalone unit test targets." >> coverage_report.md
+
+ cat coverage_report.md >> $GITHUB_STEP_SUMMARY
+ fi
+
+ END_TIME=$(date +%s)
+ echo "=== [STAGE 2/4] Targeted Tests Completed in $((END_TIME - START_TIME))s ==="
+
+ - name: Merge Profiles
+ if: env.TARGETS != ''
+ run: |
+ echo "=== [STAGE 3/4] Merging Raw Profiles at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ find build-cov -name "libc_cov_*.profraw" > profraw_list.txt
+ NUM_PROFS=$(wc -l < profraw_list.txt || echo 0)
+ echo "[LOG] Discovered $NUM_PROFS raw profile data files."
+ if [ -s profraw_list.txt ]; then
+ llvm-profdata-23 merge -sparse --input-files=profraw_list.txt -o libc_full.profdata
+ echo "[LOG] Successfully merged profile data into libc_full.profdata."
+ else
+ echo "[LOG] Warning: No profraw files found."
+ fi
+
+ - name: Extract Executables and Generate Summary
+ if: env.TARGETS != ''
+ run: |
+ echo "=== [STAGE 4/4] Generating Coverage Report at $(date -u '+%Y-%m-%d %H:%M:%S UTC') ==="
+ EXECUTABLES=($(find build-cov -type f -executable -name "*__build__"))
+ OBJECTS=("${EXECUTABLES[@]:1}")
+ OBJECTS=("${OBJECTS[@]/#/-object=}")
+
+ echo "[LOG] Discovered ${#EXECUTABLES[@]} instrumented binaries for llvm-cov export."
+
+ if [ ! -f libc_full.profdata ]; then
+ echo "[LOG] Notice: libc_full.profdata not found. Generating non-coverage summary."
+ echo "## LLVM-libc Patch Coverage Report" > coverage_report.md
+ echo "" >> coverage_report.md
+ echo "- **Base Branch:** [\`${BASE_REF}\` (${BASE_SHA:0:7})](https://github.com/${BASE_REPO}/commit/${BASE_SHA})" >> coverage_report.md
+ echo "- **Head Commit:** [\`${HEAD_REF}\` (${HEAD_SHA:0:7})](https://github.com/${HEAD_REPO}/commit/${HEAD_SHA})" >> coverage_report.md
+ echo "---" >> coverage_report.md
+ echo "" >> coverage_report.md
+ echo "**Coverage Validated**" >> coverage_report.md
+ echo "No executable statement profile data was collected for the modified lines in this patch." >> coverage_report.md
+ cat coverage_report.md >> $GITHUB_STEP_SUMMARY
+ exit 0
+ fi
+
+ # 1. Generate JSON report for Python script
+ llvm-cov-23 export -format=text -instr-profile=libc_full.profdata "${EXECUTABLES[0]}" "${OBJECTS[@]}" > coverage.json
+
+ # 2. Generate git diff for modified libc source files
+ git diff "$DIFF_BASE" HEAD -- libc/src/ > patch.diff
+
+ # 3. Run Patch Report Python Script using persisted commit metadata
+ python3 libc/utils/coverage/patch_report.py patch.diff coverage.json "$BASE_SHA" "$HEAD_SHA" "$BASE_REF" "$HEAD_REF" "$TARGETS" "$BASE_REPO" "$HEAD_REPO" > coverage_report.md
+ cat coverage_report.md >> $GITHUB_STEP_SUMMARY
+ echo "[LOG] Coverage report successfully written to GITHUB_STEP_SUMMARY."
+
+ - name: Fallback Failure Summary
+ if: failure() && !hashFiles('coverage_report.md')
+ run: |
+ echo "## LLVM-libc Patch Coverage Report" > coverage_report.md
+ echo "" >> coverage_report.md
+ echo "- **Base Branch:** [\`${BASE_REF:-main}\` (${BASE_SHA:0:7})](https://github.com/${BASE_REPO:-llvm/llvm-project}/commit/${BASE_SHA:-main})" >> coverage_report.md
+ echo "- **Head Commit:** [\`${HEAD_REF:-HEAD}\` (${HEAD_SHA:0:7})](https://github.com/${HEAD_REPO:-llvm/llvm-project}/commit/${HEAD_SHA:-HEAD})" >> coverage_report.md
+ echo "---" >> coverage_report.md
+ echo "" >> coverage_report.md
+ echo "**Coverage Validated**" >> coverage_report.md
+ echo "The targeted build or test execution encountered an error before coverage data could be finalized. Inspect the job logs for details." >> coverage_report.md
+
+ cat coverage_report.md >> $GITHUB_STEP_SUMMARY
+
+ - name: Post or Update Sticky PR Comment
+ if: always() && github.event_name == 'pull_request'
+ uses: actions/github-script at v7
+ with:
+ github-token: ${{ secrets.GITHUB_TOKEN }}
+ script: |
+ const fs = require('fs');
+ if (!fs.existsSync('coverage_report.md')) return;
+
+ const reportContent = fs.readFileSync('coverage_report.md', 'utf8');
+ const identifier = '<!-- LLVM-LIBC-PRE-COMMIT-COVERAGE-COMMENT -->';
+ const issue_number = context.payload.pull_request.number;
+ const owner = context.repo.owner;
+ const repo = context.repo.repo;
+
+ try {
+ // Fetch existing PR comments
+ const comments = await github.rest.issues.listComments({
+ owner,
+ repo,
+ issue_number,
+ per_page: 100
+ });
+
+ const botComment = comments.data.find(c => c.body && c.body.includes(identifier));
+
+ // Parse metrics from reportContent for history table entry
+ const headSha = context.payload.pull_request.head.sha.substring(0, 7);
+ const dateStr = new Date().toISOString().replace('T', ' ').substring(0, 16);
+
+ const pctMatch = reportContent.match(/\|\s*\*\*Patch Line Coverage\*\*\s*\|\s*\*\*([0-9.]+)%\*\*\s*\|/);
+ const pct = pctMatch ? pctMatch[1] + '%' : '100.00%';
+
+ const lineMatch = reportContent.match(/\|\s*\*\*Executable Lines Evaluated\*\*\s*\|\s*\*\*([0-9]+)\*\*\s*\|/);
+ const totalLines = lineMatch ? lineMatch[1] : 'N/A';
+
+ const missMatch = reportContent.match(/\|\s*\*\*Unexecuted Lines\*\*\s*\|\s*\*\*([0-9]+)\*\*\s*\|/);
+ const missedLines = missMatch ? missMatch[1] : '0';
+
+ const newHistoryRow = `| [\`${headSha}\`](https://github.com/${owner}/${repo}/commit/${context.payload.pull_request.head.sha}) | ${dateStr} | ${totalLines} | **${pct}** | ${missedLines} |`;
+
+ let existingHistory = '';
+ if (botComment && botComment.body.includes('<!-- HISTORY_START -->')) {
+ const historyStart = botComment.body.indexOf('<!-- HISTORY_START -->') + '<!-- HISTORY_START -->'.length;
+ const historyEnd = botComment.body.indexOf('<!-- HISTORY_END -->');
+ if (historyStart !== -1 && historyEnd !== -1) {
+ existingHistory = botComment.body.substring(historyStart, historyEnd).trim();
+ }
+ }
+
+ let historyTable = existingHistory;
+ if (!historyTable) {
+ historyTable = `| Commit | Date (UTC) | Executable Lines | Patch Coverage | Missed Lines |\n| :---: | :---: | :---: | :---: | :---: |\n${newHistoryRow}`;
+ } else {
+ if (!historyTable.includes(headSha)) {
+ historyTable += `\n${newHistoryRow}`;
+ }
+ }
+
+ const fullBody = `${identifier}\n${reportContent}\n\n<details>\n<summary><b>View Coverage History for this PR</b></summary>\n\n<!-- HISTORY_START -->\n${historyTable}\n<!-- HISTORY_END -->\n\n</details>`;
+
+ if (botComment) {
+ await github.rest.issues.updateComment({
+ owner,
+ repo,
+ comment_id: botComment.id,
+ body: fullBody
+ });
+ } else {
+ await github.rest.issues.createComment({
+ owner,
+ repo,
+ issue_number,
+ body: fullBody
+ });
+ }
+ } catch (err) {
+ core.warning(`Could not post/update sticky PR comment: ${err.message}`);
+ }
diff --git a/libc/utils/coverage/full_report.py b/libc/utils/coverage/full_report.py
new file mode 100644
index 0000000000000..7c2f65551fb3b
--- /dev/null
+++ b/libc/utils/coverage/full_report.py
@@ -0,0 +1,138 @@
+#!/usr/bin/env python3
+#
+# ====- Generate full codebase coverage reports ----------------*- python -*--==#
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ==-------------------------------------------------------------------------==#
+
+import argparse
+import os
+import sys
+from pathlib import Path
+from typing import Dict, List, Tuple
+
+# Ensure local module import works regardless of CWD
+sys.path.insert(0, str(Path(__file__).parent.resolve()))
+from parser import CoverageJSONParser
+
+
+def render_full_report(cov_data: dict) -> None:
+ if "data" not in cov_data or not cov_data["data"]:
+ print("## LLVM-libc Full Codebase Coverage Report\n")
+ print("> [!WARNING]")
+ print("> ### No Coverage Data Detected")
+ print("> The test execution completed but no coverage profiles were exported.")
+ return
+
+ subsystems: Dict[str, Dict[str, int]] = {}
+ file_stats: List[Tuple[str, int, int, int, int]] = []
+
+ total_lines_cov = 0
+ total_lines_tot = 0
+ total_func_cov = 0
+ total_func_tot = 0
+
+ for item in cov_data["data"][0].get("files", []):
+ fpath = item["filename"]
+ if "src/" not in fpath or "/test/" in fpath or "/utils/" in fpath:
+ continue
+
+ idx = fpath.find("src/")
+ if idx == -1:
+ continue
+ rel_path = fpath[idx:]
+
+ summary = item.get("summary", {})
+ lines_summary = summary.get("lines", {})
+ func_summary = summary.get("functions", {})
+
+ line_tot = lines_summary.get("count", 0)
+ line_cov = lines_summary.get("covered", 0)
+ func_tot = func_summary.get("count", 0)
+ func_cov = func_summary.get("covered", 0)
+
+ if line_tot == 0:
+ continue
+
+ total_lines_cov += line_cov
+ total_lines_tot += line_tot
+ total_func_cov += func_cov
+ total_func_tot += func_tot
+
+ parts = rel_path.split("/")
+ subsystem = "/".join(parts[:2]) if len(parts) >= 2 else parts[0]
+
+ if subsystem not in subsystems:
+ subsystems[subsystem] = {
+ "lines_cov": 0,
+ "lines_tot": 0,
+ "func_cov": 0,
+ "func_tot": 0,
+ }
+
+ subsystems[subsystem]["lines_cov"] += line_cov
+ subsystems[subsystem]["lines_tot"] += line_tot
+ subsystems[subsystem]["func_cov"] += func_cov
+ subsystems[subsystem]["func_tot"] += func_tot
+
+ file_stats.append((rel_path, line_cov, line_tot, func_cov, func_tot))
+
+ line_pct = (total_lines_cov / total_lines_tot * 100) if total_lines_tot > 0 else 0
+ func_pct = (total_func_cov / total_func_tot * 100) if total_func_tot > 0 else 0
+
+ repo = os.environ.get("GITHUB_REPOSITORY", "tapiwagonga/llvm-project")
+ if "/" in repo:
+ owner, repo_name = repo.split("/", 1)
+ pages_url = f"https://{owner}.github.io/{repo_name}/"
+ else:
+ pages_url = f"https://{repo}.github.io/"
+
+ print("## LLVM-libc Full Codebase Coverage Report\n")
+
+ print("> [!NOTE]")
+ print(f"> ### Overall Codebase Coverage: **{line_pct:.2f}%**")
+ print(
+ f"> Successfully tested **{total_lines_cov:,} / {total_lines_tot:,}** executable lines across all LLVM-libc subsystems."
+ )
+ print("")
+
+ print(f"- **Coverage Dashboard:** [{pages_url}]({pages_url})")
+ print("\n---\n")
+
+ print("### Codebase Health Metrics")
+ print("| Metric | Covered | Total | Coverage % |")
+ print("| :--- | :---: | :---: | :---: |")
+ print(
+ f"| **Executable Line Coverage** | {total_lines_cov:,} | {total_lines_tot:,} | **{line_pct:.2f}%** |"
+ )
+ print(
+ f"| **Function Coverage** | {total_func_cov:,} | {total_func_tot:,} | **{func_pct:.2f}%** |\n"
+ )
+
+ print("### Subsystem Coverage Breakdown")
+ print("| Subsystem | Line Coverage | Function Coverage | Executable Lines | Missed Lines |")
+ print("| :--- | :---: | :---: | :---: | :---: |")
+ for sub, data in sorted(subsystems.items()):
+ s_line_pct = (data["lines_cov"] / data["lines_tot"] * 100) if data["lines_tot"] > 0 else 0
+ s_func_pct = (data["func_cov"] / data["func_tot"] * 100) if data["func_tot"] > 0 else 0
+ missed = data["lines_tot"] - data["lines_cov"]
+ print(
+ f"| `libc/{sub}` | **{s_line_pct:.2f}%** | {s_func_pct:.2f}% | {data['lines_tot']:,} | {missed:,} |"
+ )
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="LLVM-libc Full Coverage Analyzer")
+ parser.add_argument("json_file", help="Path to llvm-cov export JSON file")
+
+ args, _ = parser.parse_known_args()
+
+ cov_data = CoverageJSONParser.load(args.json_file)
+ render_full_report(cov_data)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/libc/utils/coverage/patch_report.py b/libc/utils/coverage/patch_report.py
new file mode 100644
index 0000000000000..48f9891d4fe4b
--- /dev/null
+++ b/libc/utils/coverage/patch_report.py
@@ -0,0 +1,245 @@
+#!/usr/bin/env python3
+#
+# ====- Generate patch coverage reports ------------------------*- python -*--==#
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ==-------------------------------------------------------------------------==#
+
+import argparse
+import os
+import sys
+from pathlib import Path
+from typing import Dict, List, Optional, Set
+
+# Ensure local module import works regardless of CWD
+sys.path.insert(0, str(Path(__file__).parent.resolve()))
+from parser import CoverageJSONParser, DiffHunk, DiffParser
+
+def is_executable_line(text: str) -> bool:
+ s = text.strip()
+ if not s:
+ return False
+ # Comments
+ if (
+ s.startswith("//")
+ or s.startswith("/*")
+ or s.startswith("*")
+ or s.startswith("*/")
+ ):
+ return False
+ # Structural braces and colons
+ if s in ("{", "}", "};", "{};") or s.startswith(":"):
+ return False
+ # Preprocessor directives
+ if s.startswith("#"):
+ return False
+ # Declarations / keywords / attributes
+ if (
+ s.startswith("namespace ")
+ or s.startswith("extern ")
+ or s.startswith("using ")
+ or s.startswith("__attribute__")
+ or s.startswith("template")
+ or s.startswith("typedef ")
+ or s.startswith("struct ")
+ or s.startswith("class ")
+ or s.startswith("enum ")
+ ):
+ return False
+ return True
+
+
+def format_line_ranges(lines: Set[int]) -> str:
+ if not lines:
+ return "None"
+ sorted_lines = sorted(lines)
+ ranges = []
+ start = sorted_lines[0]
+ end = sorted_lines[0]
+ for n in sorted_lines[1:]:
+ if n == end + 1:
+ end = n
+ else:
+ ranges.append(f"`L{start}-L{end}`" if start != end else f"`L{start}`")
+ start = end = n
+ ranges.append(f"`L{start}-L{end}`" if start != end else f"`L{start}`")
+ return ", ".join(ranges)
+
+
+def render_patch_report(
+ diff_files: Dict[str, List[DiffHunk]],
+ coverage_matrix: Dict[str, Dict[str, Set[int]]],
+ base_sha: Optional[str],
+ head_sha: Optional[str],
+ base_branch: Optional[str],
+ head_branch: Optional[str],
+ targets_str: Optional[str] = None,
+ base_repo: str = "llvm/llvm-project",
+ head_repo: str = "llvm/llvm-project",
+) -> None:
+ total_covered = 0
+ total_missed = 0
+ active_files = {}
+
+ for fpath, data in coverage_matrix.items():
+ added_lines: Set[int] = set()
+ for hunk in diff_files.get(fpath, []):
+ for l_type, text, l_num in hunk.lines:
+ if l_type == "+":
+ if not is_executable_line(text):
+ continue
+ added_lines.add(l_num)
+
+ if not added_lines:
+ continue
+
+ f_covered = added_lines.intersection(data["covered"])
+ f_missed = (added_lines.intersection(data["missed"])) - f_covered
+
+ if len(data["covered"]) > 0 or len(data["missed"]) > 0:
+ total_covered += len(f_covered)
+ total_missed += len(f_missed)
+ active_files[fpath] = (f_covered, f_missed, added_lines)
+ else:
+ total_missed += len(added_lines)
+ active_files[fpath] = (set(), added_lines, added_lines)
+
+ total_lines = total_covered + total_missed
+
+ if total_lines == 0 or not active_files:
+ print("## LLVM-libc Patch Coverage Report\n")
+ if base_sha and head_sha and base_branch and head_branch:
+ print(
+ f"- **Base Branch:** [`{base_branch}` ({base_sha[:7]})](https://github.com/{base_repo}/commit/{base_sha})"
+ )
+ print(
+ f"- **Head Commit:** [`{head_branch}` ({head_sha[:7]})](https://github.com/{head_repo}/commit/{head_sha})\n"
+ )
+ print("---\n")
+ print("> [!NOTE]")
+ print("> ### Coverage Validated")
+ print("> No `.cpp` source files in `libc/src/` were modified in this patch.")
+ sys.exit(0)
+
+ print("## LLVM-libc Patch Coverage Report\n")
+
+ coverage_percent = (total_covered / total_lines) * 100
+
+ # Modern GitHub UI Alert Card
+ if total_missed == 0:
+ print("> [!TIP]")
+ print(f"> ### Patch Coverage: **{coverage_percent:.2f}%** (PASSED)")
+ print(
+ f"> All **{total_lines}** newly added or modified executable lines are covered by targeted unit tests."
+ )
+ else:
+ print("> [!WARNING]")
+ print(f"> ### Patch Coverage: **{coverage_percent:.2f}%** ({total_missed} Missed Lines)")
+ print(
+ f"> **{total_missed}** unexecuted line(s) detected in your patch. Please review the missing lines below."
+ )
+ print("")
+
+ # Commit metadata and targets executed with exact upstream and fork repositories
+ if base_sha and head_sha and base_branch and head_branch:
+ print(
+ f"- **Base Branch:** [`{base_branch}` ({base_sha[:7]})](https://github.com/{base_repo}/commit/{base_sha})"
+ )
+ print(
+ f"- **Head Commit:** [`{head_branch}` ({head_sha[:7]})](https://github.com/{head_repo}/commit/{head_sha})"
+ )
+ if targets_str:
+ targets_formatted = ", ".join(f"`{t.strip()}`" for t in targets_str.split() if t.strip())
+ print(f"- **Targeted Tests Executed:** {targets_formatted}")
+ print("\n---\n")
+
+ # Executive Summary Table
+ status_label = "**PASSED**" if total_missed == 0 else "**ACTION REQUIRED**"
+ commit_link = f"[`{head_sha[:7]}`](https://github.com/{head_repo}/commit/{head_sha})" if head_sha else "HEAD"
+
+ print("### Executive Summary")
+ print(f"The code coverage on the recent commit {commit_link} is **{coverage_percent:.2f}%**.")
+ print("")
+ print("| Metric | Value | Status |")
+ print("| :--- | :---: | :---: |")
+ print(f"| **Patch Line Coverage** | **{coverage_percent:.2f}%** | {status_label} |")
+ print(f"| **Executable Lines Evaluated** | **{total_lines}** | — |")
+ print(f"| **Covered Lines** | **{total_covered}** | {coverage_percent:.1f}% |")
+ print(f"| **Unexecuted Lines** | **{total_missed}** | {'0' if total_missed == 0 else str(total_missed)} |")
+ print("")
+
+ # Modified Files Impact Table
+ print("### Modified Files Impact")
+ print("| Modified Source File | Patch Coverage | Covered / Total | Missed Lines | Unexecuted Line Spans |")
+ print("| :--- | :---: | :---: | :---: | :---: |")
+
+ for fpath, (f_covered, f_missed, added_lines) in active_files.items():
+ f_total = len(f_covered) + len(f_missed)
+ f_pct = (len(f_covered) / f_total * 100) if f_total > 0 else 0.0
+ line_spans = format_line_ranges(f_missed)
+ file_link = f"[`{fpath}`](https://github.com/{head_repo}/blob/{head_sha or 'main'}/{fpath})"
+ print(
+ f"| {file_link} | **{f_pct:.2f}%** | {len(f_covered)} / {f_total} | {len(f_missed)} | {line_spans} |"
+ )
+ print("")
+
+ # Collapsible Source Map Diff
+ print("<details>")
+ print("<summary><b>View Annotated Patch Diff (Source Map)</b></summary>\n")
+
+ for fpath, (f_covered, f_missed, added_lines) in active_files.items():
+ hunks = diff_files.get(fpath, [])
+ print(f"#### `{fpath}`")
+ print("```diff")
+ for hunk in hunks:
+ print(hunk.header)
+ for l_type, text, l_num in hunk.lines:
+ if l_type == "+":
+ if l_num in f_missed:
+ print(f"- {text}")
+ elif l_num in f_covered:
+ print(f"+ {text}")
+ else:
+ print(f" {text}")
+ elif l_type == " ":
+ print(f" {text}")
+ print("```\n")
+ print("</details>")
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="LLVM-libc Patch Coverage Analyzer")
+ parser.add_argument("diff_file", help="Path to unified diff file")
+ parser.add_argument("json_file", help="Path to llvm-cov export JSON file")
+ parser.add_argument("base_sha", nargs="?", help="Base commit SHA")
+ parser.add_argument("head_sha", nargs="?", help="Head commit SHA")
+ parser.add_argument("base_branch", nargs="?", help="Base branch name")
+ parser.add_argument("head_branch", nargs="?", help="Head branch name")
+ parser.add_argument("targets", nargs="?", help="Space-separated list of executed test targets")
+ parser.add_argument("base_repo", nargs="?", default="llvm/llvm-project", help="Base repository (e.g. llvm/llvm-project)")
+ parser.add_argument("head_repo", nargs="?", default="llvm/llvm-project", help="Head repository (e.g. tapiwagonga/llvm-project)")
+
+ args = parser.parse_args()
+
+ diff_files = DiffParser.parse(args.diff_file)
+ cov_data = CoverageJSONParser.load(args.json_file)
+ coverage_matrix = CoverageJSONParser.extract_patch_matrix(cov_data, diff_files)
+
+ render_patch_report(
+ diff_files,
+ coverage_matrix,
+ args.base_sha,
+ args.head_sha,
+ args.base_branch,
+ args.head_branch,
+ args.targets,
+ args.base_repo,
+ args.head_repo,
+ )
+
+
+if __name__ == "__main__":
+ main()
More information about the libc-commits
mailing list