[libcxx-commits] [libcxx] [libcxx][docs] Make test name pattern documentation more obvious (PR #73136)
Will Hawkins via libcxx-commits
libcxx-commits at lists.llvm.org
Wed Dec 13 19:13:00 PST 2023
https://github.com/hawkinsw updated https://github.com/llvm/llvm-project/pull/73136
>From b46ef9e42a525433eeb3959b0c92468120e63ae4 Mon Sep 17 00:00:00 2001
From: Will Hawkins <hawkinsw at obs.cr>
Date: Wed, 22 Nov 2023 10:12:04 -0500
Subject: [PATCH 1/5] [libcxx][docs] Make test name pattern documentation more
obvious
As a new contributor, I found it hard to find the documentation for the
meaning of the names of different tests and how those names translate
to Lit.
This makes that documentation more explicit.
---
libcxx/docs/TestingLibcxx.rst | 49 ++++++++++++++++++++++++++++++-----
1 file changed, 42 insertions(+), 7 deletions(-)
diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst
index 44f3904f4e426a..5783cc6a21922c 100644
--- a/libcxx/docs/TestingLibcxx.rst
+++ b/libcxx/docs/TestingLibcxx.rst
@@ -330,13 +330,48 @@ additional headers.
taken to make it work in earlier language versions.
-Additional reading
-------------------
-
-The function ``CxxStandardLibraryTest`` in the file
-``libcxx/utils/libcxx/test/format.py`` has documentation about writing test. It
-explains the difference between the test named ``foo.pass.cpp`` and named
-``foo.verify.cpp`` are.
+Test names
+----------
+
+The names of test files have meaning for the libcxx-specific configuration of
+Lit. A complete description can be found in the docstring for the function
+``CxxStandardLibraryTest`` in the file ``libcxx/utils/libcxx/test/format.py``.
+
+For quick reference, here is an overview of some of the most useful patterns:
+
+.. list-table:: Lit Meaning of libcxx Test Filenames
+ :widths: 25 75
+ :header-rows: 1
+
+ * - Name Pattern
+ - Meaning
+ * - ``FOO.pass.cpp``
+ - Compiles, links and runs successfully
+ * - ``FOO.pass.mm``
+ - Same as ``FOO.pass.cpp``, but for Objective-C++
+
+ * - ``FOO.compile.pass.cpp``
+ - Compiles successfully, link and run not attempted
+ * - ``FOO.compile.pass.mm``
+ - Same as ``FOO.compile.pass.cpp``, but for Objective-C++
+ * - ``FOO.compile.fail.cpp``
+ - Does not compile successfully
+
+ * - ``FOO.link.pass.cpp``
+ - Compiles and links successfully, run not attempted
+ * - ``FOO.link.pass.mm``
+ - Same as ``FOO.link.pass.cpp``, but for Objective-C++
+ * - ``FOO.link.fail.cpp``
+ - Compiles successfully, but fails to link
+
+ * - ``FOO.sh.<anything>``
+ - A builtin Lit Shell test
+
+ * - ``FOO.gen.<anything>``
+ - A ``.sh`` test that generates one or more Lit tests on the fly. Executing this test must generate one or more files as expected by LLVM split-file, and each generated file leads to a separate Lit test that runs that file as defined by the test format. This can be used to generate multiple Lit tests from a single source file, which is useful for testing repetitive properties in the library. Be careful not to abuse this since this is not a replacement for usual code reuse techniques.
+
+ * - ``FOO.verify.cpp``
+ - Compiles with clang-verify. This type of test is automatically marked as UNSUPPORTED if the compiler does not support clang-verify.
Benchmarks
==========
>From c148c822407d5ee402364a88dd55bb503fa1b67c Mon Sep 17 00:00:00 2001
From: Will Hawkins <hawkinsw at obs.cr>
Date: Wed, 22 Nov 2023 21:21:02 -0500
Subject: [PATCH 2/5] fixup! [libcxx][docs] Make test name pattern
documentation more obvious
Update to consolidate lit test file name pattern semantic documentation to docs/
---
libcxx/utils/libcxx/test/format.py | 36 ++++--------------------------
1 file changed, 4 insertions(+), 32 deletions(-)
diff --git a/libcxx/utils/libcxx/test/format.py b/libcxx/utils/libcxx/test/format.py
index 52c6f9cd8f2ef2..dc5291a62464aa 100644
--- a/libcxx/utils/libcxx/test/format.py
+++ b/libcxx/utils/libcxx/test/format.py
@@ -155,38 +155,10 @@ class CxxStandardLibraryTest(lit.formats.FileBasedTest):
"""
Lit test format for the C++ Standard Library conformance test suite.
- This test format is based on top of the ShTest format -- it basically
- creates a shell script performing the right operations (compile/link/run)
- based on the extension of the test file it encounters. It supports files
- with the following extensions:
-
- FOO.pass.cpp - Compiles, links and runs successfully
- FOO.pass.mm - Same as .pass.cpp, but for Objective-C++
-
- FOO.compile.pass.cpp - Compiles successfully, link and run not attempted
- FOO.compile.pass.mm - Same as .compile.pass.cpp, but for Objective-C++
- FOO.compile.fail.cpp - Does not compile successfully
-
- FOO.link.pass.cpp - Compiles and links successfully, run not attempted
- FOO.link.pass.mm - Same as .link.pass.cpp, but for Objective-C++
- FOO.link.fail.cpp - Compiles successfully, but fails to link
-
- FOO.sh.<anything> - A builtin Lit Shell test
-
- FOO.gen.<anything> - A .sh test that generates one or more Lit tests on the
- fly. Executing this test must generate one or more files
- as expected by LLVM split-file, and each generated file
- leads to a separate Lit test that runs that file as
- defined by the test format. This can be used to generate
- multiple Lit tests from a single source file, which is
- useful for testing repetitive properties in the library.
- Be careful not to abuse this since this is not a replacement
- for usual code reuse techniques.
-
- FOO.verify.cpp - Compiles with clang-verify. This type of test is
- automatically marked as UNSUPPORTED if the compiler
- does not support Clang-verify.
-
+ Lit tests are contained in files that follow a certain pattern. That
+ pattern determines the semantics of the test. See
+ libcxx/docs/TestingLibcxx.rst or https://libcxx.llvm.org/TestingLibcxx.html
+ for a complete description of those semantics.
Substitution requirements
===============================
>From 039de0768842ec9dad0870f731885e6e7770c2bf Mon Sep 17 00:00:00 2001
From: Will Hawkins <hawkinsw at obs.cr>
Date: Mon, 27 Nov 2023 09:15:47 -0500
Subject: [PATCH 3/5] fixup! [libcxx][docs] Make test name pattern
documentation more obvious
(Fully) centralize all naming documentation in the rst-based documentation.
---
libcxx/docs/TestingLibcxx.rst | 41 ++++++++++++++++++++---------------
1 file changed, 23 insertions(+), 18 deletions(-)
diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst
index 5783cc6a21922c..b93f61388cda6c 100644
--- a/libcxx/docs/TestingLibcxx.rst
+++ b/libcxx/docs/TestingLibcxx.rst
@@ -334,11 +334,12 @@ Test names
----------
The names of test files have meaning for the libcxx-specific configuration of
-Lit. A complete description can be found in the docstring for the function
-``CxxStandardLibraryTest`` in the file ``libcxx/utils/libcxx/test/format.py``.
-
-For quick reference, here is an overview of some of the most useful patterns:
+Lit. Based on the pattern that matches the name of a test file, Lit will test
+the code contained therein in different ways. Refer to the `Lit Meaning of libcxx
+Test Filenames`_ when
+determining the names for new test files.
+.. _Lit Meaning of libcxx Test Filenames:
.. list-table:: Lit Meaning of libcxx Test Filenames
:widths: 25 75
:header-rows: 1
@@ -346,32 +347,36 @@ For quick reference, here is an overview of some of the most useful patterns:
* - Name Pattern
- Meaning
* - ``FOO.pass.cpp``
- - Compiles, links and runs successfully
+ - Checks whether the C++ code in the file compiles, links and runs successfully.
* - ``FOO.pass.mm``
- - Same as ``FOO.pass.cpp``, but for Objective-C++
+ - Same as ``FOO.pass.cpp``, but for Objective-C++.
* - ``FOO.compile.pass.cpp``
- - Compiles successfully, link and run not attempted
+ - Checks whether the C++ code in the file compiles successfully.
* - ``FOO.compile.pass.mm``
- - Same as ``FOO.compile.pass.cpp``, but for Objective-C++
+ - Same as ``FOO.compile.pass.cpp``, but for Objective-C++.
* - ``FOO.compile.fail.cpp``
- - Does not compile successfully
+ - Checks that the code in the file does *not* compile successfully.
+
+ * - ``FOO.verify.cpp``
+ - Compiles with clang-verify. This type of test is automatically marked as UNSUPPORTED if the compiler does not support clang-verify.
* - ``FOO.link.pass.cpp``
- - Compiles and links successfully, run not attempted
+ - Checks that the C++ code in the file compiles and links successfully -- no run attempted.
* - ``FOO.link.pass.mm``
- - Same as ``FOO.link.pass.cpp``, but for Objective-C++
+ - Same as ``FOO.link.pass.cpp``, but for Objective-C++.
* - ``FOO.link.fail.cpp``
- - Compiles successfully, but fails to link
+ - Checks whether the C++ code in the file fails to link after successful compilation.
+ * - ``FOO.link.fail.mm``
+ - Same as ``FOO.link.fail.cpp``, but for Objective-C++.
* - ``FOO.sh.<anything>``
- - A builtin Lit Shell test
-
+ - A *builtin Lit Shell* test.
* - ``FOO.gen.<anything>``
- - A ``.sh`` test that generates one or more Lit tests on the fly. Executing this test must generate one or more files as expected by LLVM split-file, and each generated file leads to a separate Lit test that runs that file as defined by the test format. This can be used to generate multiple Lit tests from a single source file, which is useful for testing repetitive properties in the library. Be careful not to abuse this since this is not a replacement for usual code reuse techniques.
-
- * - ``FOO.verify.cpp``
- - Compiles with clang-verify. This type of test is automatically marked as UNSUPPORTED if the compiler does not support clang-verify.
+ - A *builtin Lit Shell* test that generates one or more Lit tests on the fly. Executing this test must generate one or more files as expected
+ by LLVM split-file. Each generated file will drive an invocation of a separate Lit test. The format of the generated file will determine the type
+ of Lit test to be executed. This can be used to generate multiple Lit tests from a single source file, which is useful for testing repetitive properties
+ in the library. Be careful not to abuse this since this is not a replacement for usual code reuse techniques.
Benchmarks
==========
>From 3c1297b30674c8854a0a1ae7b7e0d11f384ec391 Mon Sep 17 00:00:00 2001
From: Will Hawkins <hawkinsw at obs.cr>
Date: Mon, 27 Nov 2023 09:19:29 -0500
Subject: [PATCH 4/5] fixup! [libcxx][docs] Make test name pattern
documentation more obvious
Point only to www documentation for test names (per @mordante's excellent feedback)
---
libcxx/utils/libcxx/test/format.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/libcxx/utils/libcxx/test/format.py b/libcxx/utils/libcxx/test/format.py
index dc5291a62464aa..cd4807f1855738 100644
--- a/libcxx/utils/libcxx/test/format.py
+++ b/libcxx/utils/libcxx/test/format.py
@@ -157,7 +157,7 @@ class CxxStandardLibraryTest(lit.formats.FileBasedTest):
Lit tests are contained in files that follow a certain pattern. That
pattern determines the semantics of the test. See
- libcxx/docs/TestingLibcxx.rst or https://libcxx.llvm.org/TestingLibcxx.html
+ https://libcxx.llvm.org/TestingLibcxx.html#test-names
for a complete description of those semantics.
Substitution requirements
>From 0d8d731a36a382400df53df004861d61ac38009a Mon Sep 17 00:00:00 2001
From: Will Hawkins <whh8b at obs.cr>
Date: Wed, 13 Dec 2023 22:12:52 -0500
Subject: [PATCH 5/5] Update libcxx/docs/TestingLibcxx.rst
Be more clear about what type of shell we get when running these tests.
Co-authored-by: Louis Dionne <ldionne.2 at gmail.com>
---
libcxx/docs/TestingLibcxx.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst
index b93f61388cda6c..5ccac803f793cb 100644
--- a/libcxx/docs/TestingLibcxx.rst
+++ b/libcxx/docs/TestingLibcxx.rst
@@ -373,7 +373,7 @@ determining the names for new test files.
* - ``FOO.sh.<anything>``
- A *builtin Lit Shell* test.
* - ``FOO.gen.<anything>``
- - A *builtin Lit Shell* test that generates one or more Lit tests on the fly. Executing this test must generate one or more files as expected
+ - A variant of a *Lit Shell* test that generates one or more Lit tests on the fly. Executing this test must generate one or more files as expected
by LLVM split-file. Each generated file will drive an invocation of a separate Lit test. The format of the generated file will determine the type
of Lit test to be executed. This can be used to generate multiple Lit tests from a single source file, which is useful for testing repetitive properties
in the library. Be careful not to abuse this since this is not a replacement for usual code reuse techniques.
More information about the libcxx-commits
mailing list