[libcxx] [llvm] [libc++] Optionally support filecheck-based tests (PR #165769)

[via llvm-commits]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-libcxx

Author: Louis Dionne (ldionne)

<details>
<summary>Changes</summary>

This patch adds support for filecheck tests in libc++'s test suite. However, it doesn't use LLVM's filecheck program, which requires building a bunch of LLVM utilities. Instead, it installs the Python port of filecheck at https://github.com/AntonLydike/filecheck which supports basically the same functionality.

The test suite still works when filecheck is not available, since tests that use filecheck should be guarded on the `has-filecheck` Lit feature.

This should make it possible to test several things that were previously impossible to test, especially for specific code generation.

---
Full diff: https://github.com/llvm/llvm-project/pull/165769.diff


6 Files Affected:

- (modified) .github/workflows/libcxx-build-and-test.yaml (+1-1) 
- (modified) libcxx/docs/TestingLibcxx.rst (+7) 
- (added) libcxx/test/requirements.txt (+5) 
- (added) libcxx/test/selftest/filecheck.sh.cpp (+15) 
- (modified) libcxx/utils/ci/Dockerfile (+3) 
- (modified) libcxx/utils/libcxx/test/features.py (+8) 


``````````diff
diff --git a/.github/workflows/libcxx-build-and-test.yaml b/.github/workflows/libcxx-build-and-test.yaml
index 6c8f2cb45ee0a..6097174f5e553 100644
--- a/.github/workflows/libcxx-build-and-test.yaml
+++ b/.github/workflows/libcxx-build-and-test.yaml
@@ -221,7 +221,7 @@ jobs:
         run: |
           python3 -m venv .venv
           source .venv/bin/activate
-          python -m pip install psutil
+          pip install -r libcxx/test/requirements.txt
           bash libcxx/utils/ci/run-buildbot ${{ matrix.config }}
       - uses: actions/upload-artifact at ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
         if: always()  # Upload artifacts even if the build or test suite fails
diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst
index dbe69484abedf..d77c1c80fd307 100644
--- a/libcxx/docs/TestingLibcxx.rst
+++ b/libcxx/docs/TestingLibcxx.rst
@@ -23,6 +23,13 @@ Please see the `Lit Command Guide`_ for more information about LIT.
 
 .. _LIT Command Guide: https://llvm.org/docs/CommandGuide/lit.html
 
+Dependencies
+------------
+
+The libc++ test suite has a few optional dependencies. These can be installed
+with ``pip install -r libcxx/test/requirements.txt``. Installing these dependencies
+will ensure that the maximum number of tests can be run.
+
 Usage
 -----
 
diff --git a/libcxx/test/requirements.txt b/libcxx/test/requirements.txt
new file mode 100644
index 0000000000000..842b8ca4ef901
--- /dev/null
+++ b/libcxx/test/requirements.txt
@@ -0,0 +1,5 @@
+#
+# This file defines Python requirements to run the libc++ test suite.
+#
+filecheck
+psutil
diff --git a/libcxx/test/selftest/filecheck.sh.cpp b/libcxx/test/selftest/filecheck.sh.cpp
new file mode 100644
index 0000000000000..911d22b414b3b
--- /dev/null
+++ b/libcxx/test/selftest/filecheck.sh.cpp
@@ -0,0 +1,15 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: has-filecheck
+
+// Make sure that we can use filecheck to write tests when the `has-filecheck`
+// Lit feature is defined.
+
+// RUN: echo "hello world" | filecheck %s
+// CHECK: hello world
diff --git a/libcxx/utils/ci/Dockerfile b/libcxx/utils/ci/Dockerfile
index d22deec4dadab..40592c3770c76 100644
--- a/libcxx/utils/ci/Dockerfile
+++ b/libcxx/utils/ci/Dockerfile
@@ -111,6 +111,9 @@ RUN sudo apt-get update \
         xz-utils \
     && sudo rm -rf /var/lib/apt/lists/*
 
+COPY ../../test/requirements.txt .
+RUN python3 -m pip install -r requirements.txt
+
 RUN <<EOF
   set -e
   wget -qO /tmp/ninja.gz https://github.com/ninja-build/ninja/releases/latest/download/ninja-linux.zip
diff --git a/libcxx/utils/libcxx/test/features.py b/libcxx/utils/libcxx/test/features.py
index 7d6e78de343c5..21e86a44e745b 100644
--- a/libcxx/utils/libcxx/test/features.py
+++ b/libcxx/utils/libcxx/test/features.py
@@ -355,6 +355,14 @@ def _mingwSupportsModules(cfg):
         name="has-no-zdump",
         when=lambda cfg: runScriptExitCode(cfg, ["zdump --version"]) != 0,
     ),
+    # Whether the `filecheck` executable is available. Note that this corresponds to
+    # a Python port of LLVM's FileCheck, not LLVM's actual FileCheck program, since
+    # that one requires building parts of LLVM that we don't want to build when merely
+    # testing libc++.
+    Feature(
+        name="has-filecheck",
+        when=lambda cfg: runScriptExitCode(cfg, ["filecheck --version"]) == 0,
+    ),
 ]
 
 # Deduce and add the test features that that are implied by the #defines in

``````````

</details>


https://github.com/llvm/llvm-project/pull/165769