[libc-commits] [libc] [libc] Extend check-libc-lit to cover include, integration, and all src tests (PR #184366)

[Jeff Bailey via libc-commits]

https://github.com/kaladron created https://github.com/llvm/llvm-project/pull/184366

The lit-based test runner introduced in c776a52f only discovered libc.test.src tests with a strict __unit__.__build__ or __hermetic__.__build__ suffix. This missed four categories of tests:

1. libc.test.include.* tests (e.g. isnan_test, signbit_test)
2. libc.test.integration.* tests (e.g. pthread, unistd, startup)
3. libc.test.src.* tests that have no __unit__/__hermetic__ marker (e.g. errno_test, dirent_test, htonl)
4. libc.test.src.* tests with extra option suffixes between the type marker and .__build__ (e.g. __unit__.__NO_FMA_OPT.__build__)

Wire up the two missing build dependencies so that check-libc-lit builds include and integration tests before running them, and update _isTestExecutable() to recognise all four patterns.

The pattern documentation was consolidated into the _isTestExecutable() docstring, where it is next to the code it describes, to avoid the two diverging in future.

Tested:
Compared the test count from a full `ninja check-libc` run (2765 tests) against `llvm-lit --show-tests libc/test` after this change and confirmed the counts match exactly.

>From 1fd904e986833787ca6c1e05cfa95ef197e5a4a6 Mon Sep 17 00:00:00 2001
From: Jeff Bailey <jeffbailey at google.com>
Date: Tue, 3 Mar 2026 15:39:42 +0000
Subject: [PATCH] [libc] Extend check-libc-lit to cover include, integration,
 and all src tests

The lit-based test runner introduced in c776a52f only discovered
libc.test.src tests with a strict __unit__.__build__ or
__hermetic__.__build__ suffix. This missed four categories of tests:

1. libc.test.include.* tests (e.g. isnan_test, signbit_test)
2. libc.test.integration.* tests (e.g. pthread, unistd, startup)
3. libc.test.src.* tests that have no __unit__/__hermetic__ marker
   (e.g. errno_test, dirent_test, htonl)
4. libc.test.src.* tests with extra option suffixes between the type
   marker and .__build__ (e.g. __unit__.__NO_FMA_OPT.__build__)

Wire up the two missing build dependencies so that check-libc-lit
builds include and integration tests before running them, and update
_isTestExecutable() to recognise all four patterns.

The pattern documentation was consolidated into the _isTestExecutable()
docstring, where it is next to the code it describes, to avoid the two
diverging in future.

Tested:
Compared the test count from a full `ninja check-libc` run
(2765 tests) against `llvm-lit --show-tests libc/test` after this
change and confirmed the counts match exactly.
---
 libc/test/include/CMakeLists.txt     |  1 +
 libc/test/integration/CMakeLists.txt |  1 +
 libc/utils/libctest/format.py        | 40 ++++++++++++++++------------
 3 files changed, 25 insertions(+), 17 deletions(-)

diff --git a/libc/test/include/CMakeLists.txt b/libc/test/include/CMakeLists.txt
index 3ac5615d7e209..c1828f51c261e 100644
--- a/libc/test/include/CMakeLists.txt
+++ b/libc/test/include/CMakeLists.txt
@@ -1,5 +1,6 @@
 add_custom_target(libc_include_tests)
 add_dependencies(check-libc libc_include_tests)
+add_dependencies(check-libc-lit libc_include_tests)
 
 add_libc_test(
   assert_test
diff --git a/libc/test/integration/CMakeLists.txt b/libc/test/integration/CMakeLists.txt
index 7dbcc9544a0b9..9a7e9a57752e6 100644
--- a/libc/test/integration/CMakeLists.txt
+++ b/libc/test/integration/CMakeLists.txt
@@ -1,5 +1,6 @@
 add_custom_target(libc-integration-tests)
 add_dependencies(check-libc libc-integration-tests)
+add_dependencies(check-libc-lit libc-integration-tests)
 
 function(add_libc_integration_test_suite name)
   add_custom_target(${name})
diff --git a/libc/utils/libctest/format.py b/libc/utils/libctest/format.py
index df7d402799762..82f65586e8b3b 100644
--- a/libc/utils/libctest/format.py
+++ b/libc/utils/libctest/format.py
@@ -15,11 +15,7 @@
 The lit config sets test_source_root == test_exec_root (both to the build
 directory), following the pattern used by llvm/test/Unit/lit.cfg.py.
 
-Test executables are discovered by looking for files matching:
-  libc.test.src.<category>.<test_name>.__unit__.__build__
-  libc.test.src.<category>.<test_name>.__hermetic__.__build__
-
-These are created by the add_libc_test() infrastructure.
+Test executables are discovered by _isTestExecutable() and run by execute().
 """
 
 import os
@@ -34,10 +30,8 @@ class LibcTest(lit.formats.ExecutableTest):
     """
     Test format for libc unit tests.
 
-    Extends ExecutableTest to discover tests from the build directory
-    rather than the source directory. Test executables are named like:
-      libc.test.src.ctype.isalnum_test.__unit__.__build__
-    and return 0 on success.
+    Extends ExecutableTest to discover pre-built test executables in the
+    build directory rather than the source directory.
     """
 
     def getTestsInDirectory(self, testSuite, path_in_suite, litConfig, localConfig):
@@ -63,16 +57,28 @@ def getTestsInDirectory(self, testSuite, path_in_suite, litConfig, localConfig):
                 yield lit.Test.Test(testSuite, path_in_suite + (filename,), localConfig)
 
     def _isTestExecutable(self, filename, filepath):
-        """Check if a file is a test executable we should run."""
-        # Pattern: libc.test.src.*.__unit__.__build__ or .__hermetic__.__build__
-        if not filename.startswith("libc.test."):
+        """
+        Check if a file is a libc test executable we should run.
+
+        Recognized patterns (all must end with .__build__):
+          libc.test.src.<category>.<test_name>.__build__
+          libc.test.src.<category>.<test_name>.__unit__[.<opts>...].__build__
+          libc.test.src.<category>.<test_name>.__hermetic__[.<opts>...].__build__
+          libc.test.include.<test_name>.__unit__[.<opts>...].__build__
+          libc.test.include.<test_name>.__hermetic__[.<opts>...].__build__
+          libc.test.integration.<category>.<test_name>.__build__
+        """
+        if not filename.endswith(".__build__"):
             return False
-        if not (
-            filename.endswith(".__unit__.__build__")
-            or filename.endswith(".__hermetic__.__build__")
-        ):
+        if filename.startswith("libc.test.src."):
+            pass  # Accept all src tests ending in .__build__
+        elif filename.startswith("libc.test.include."):
+            if ".__unit__." not in filename and ".__hermetic__." not in filename:
+                return False
+        elif filename.startswith("libc.test.integration."):
+            pass  # Accept all integration tests ending in .__build__
+        else:
             return False
-        # Must be executable
         if not os.path.isfile(filepath):
             return False
         if not os.access(filepath, os.X_OK):