[libcxx-commits] [libcxx] [libcxx][docs] Make test name pattern documentation more obvious (PR #73136)
Will Hawkins via libcxx-commits
libcxx-commits at lists.llvm.org
Fri Dec 29 17:17:19 PST 2023
https://github.com/hawkinsw updated https://github.com/llvm/llvm-project/pull/73136
>From c3d4e8724e4b707e7180aa3517f2b9b89424e555 Mon Sep 17 00:00:00 2001
From: Will Hawkins <hawkinsw at obs.cr>
Date: Wed, 22 Nov 2023 10:12:04 -0500
Subject: [PATCH 01/10] [libcxx][docs] Make test name pattern documentation
more obvious
As a new contributor, I found it hard to find the documentation for the
meaning of the names of different tests and how those names translate
to Lit.
This makes that documentation more explicit.
---
libcxx/docs/TestingLibcxx.rst | 49 ++++++++++++++++++++++++++++++-----
1 file changed, 42 insertions(+), 7 deletions(-)
diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst
index 44f3904f4e426a..5783cc6a21922c 100644
--- a/libcxx/docs/TestingLibcxx.rst
+++ b/libcxx/docs/TestingLibcxx.rst
@@ -330,13 +330,48 @@ additional headers.
taken to make it work in earlier language versions.
-Additional reading
-------------------
-
-The function ``CxxStandardLibraryTest`` in the file
-``libcxx/utils/libcxx/test/format.py`` has documentation about writing test. It
-explains the difference between the test named ``foo.pass.cpp`` and named
-``foo.verify.cpp`` are.
+Test names
+----------
+
+The names of test files have meaning for the libcxx-specific configuration of
+Lit. A complete description can be found in the docstring for the function
+``CxxStandardLibraryTest`` in the file ``libcxx/utils/libcxx/test/format.py``.
+
+For quick reference, here is an overview of some of the most useful patterns:
+
+.. list-table:: Lit Meaning of libcxx Test Filenames
+ :widths: 25 75
+ :header-rows: 1
+
+ * - Name Pattern
+ - Meaning
+ * - ``FOO.pass.cpp``
+ - Compiles, links and runs successfully
+ * - ``FOO.pass.mm``
+ - Same as ``FOO.pass.cpp``, but for Objective-C++
+
+ * - ``FOO.compile.pass.cpp``
+ - Compiles successfully, link and run not attempted
+ * - ``FOO.compile.pass.mm``
+ - Same as ``FOO.compile.pass.cpp``, but for Objective-C++
+ * - ``FOO.compile.fail.cpp``
+ - Does not compile successfully
+
+ * - ``FOO.link.pass.cpp``
+ - Compiles and links successfully, run not attempted
+ * - ``FOO.link.pass.mm``
+ - Same as ``FOO.link.pass.cpp``, but for Objective-C++
+ * - ``FOO.link.fail.cpp``
+ - Compiles successfully, but fails to link
+
+ * - ``FOO.sh.<anything>``
+ - A builtin Lit Shell test
+
+ * - ``FOO.gen.<anything>``
+ - A ``.sh`` test that generates one or more Lit tests on the fly. Executing this test must generate one or more files as expected by LLVM split-file, and each generated file leads to a separate Lit test that runs that file as defined by the test format. This can be used to generate multiple Lit tests from a single source file, which is useful for testing repetitive properties in the library. Be careful not to abuse this since this is not a replacement for usual code reuse techniques.
+
+ * - ``FOO.verify.cpp``
+ - Compiles with clang-verify. This type of test is automatically marked as UNSUPPORTED if the compiler does not support clang-verify.
Benchmarks
==========
>From ec88b9f26f3708422384ecc8288b23965f48a760 Mon Sep 17 00:00:00 2001
From: Will Hawkins <hawkinsw at obs.cr>
Date: Wed, 22 Nov 2023 21:21:02 -0500
Subject: [PATCH 02/10] fixup! [libcxx][docs] Make test name pattern
documentation more obvious
Update to consolidate lit test file name pattern semantic documentation to docs/
---
libcxx/utils/libcxx/test/format.py | 36 ++++--------------------------
1 file changed, 4 insertions(+), 32 deletions(-)
diff --git a/libcxx/utils/libcxx/test/format.py b/libcxx/utils/libcxx/test/format.py
index e58e404bfcd2a5..ace208c98459de 100644
--- a/libcxx/utils/libcxx/test/format.py
+++ b/libcxx/utils/libcxx/test/format.py
@@ -151,38 +151,10 @@ class CxxStandardLibraryTest(lit.formats.FileBasedTest):
"""
Lit test format for the C++ Standard Library conformance test suite.
- This test format is based on top of the ShTest format -- it basically
- creates a shell script performing the right operations (compile/link/run)
- based on the extension of the test file it encounters. It supports files
- with the following extensions:
-
- FOO.pass.cpp - Compiles, links and runs successfully
- FOO.pass.mm - Same as .pass.cpp, but for Objective-C++
-
- FOO.compile.pass.cpp - Compiles successfully, link and run not attempted
- FOO.compile.pass.mm - Same as .compile.pass.cpp, but for Objective-C++
- FOO.compile.fail.cpp - Does not compile successfully
-
- FOO.link.pass.cpp - Compiles and links successfully, run not attempted
- FOO.link.pass.mm - Same as .link.pass.cpp, but for Objective-C++
- FOO.link.fail.cpp - Compiles successfully, but fails to link
-
- FOO.sh.<anything> - A builtin Lit Shell test
-
- FOO.gen.<anything> - A .sh test that generates one or more Lit tests on the
- fly. Executing this test must generate one or more files
- as expected by LLVM split-file, and each generated file
- leads to a separate Lit test that runs that file as
- defined by the test format. This can be used to generate
- multiple Lit tests from a single source file, which is
- useful for testing repetitive properties in the library.
- Be careful not to abuse this since this is not a replacement
- for usual code reuse techniques.
-
- FOO.verify.cpp - Compiles with clang-verify. This type of test is
- automatically marked as UNSUPPORTED if the compiler
- does not support Clang-verify.
-
+ Lit tests are contained in files that follow a certain pattern. That
+ pattern determines the semantics of the test. See
+ libcxx/docs/TestingLibcxx.rst or https://libcxx.llvm.org/TestingLibcxx.html
+ for a complete description of those semantics.
Substitution requirements
===============================
>From 7b819a7637ca246df208ba43ba72ec68a5516c12 Mon Sep 17 00:00:00 2001
From: Will Hawkins <hawkinsw at obs.cr>
Date: Mon, 27 Nov 2023 09:15:47 -0500
Subject: [PATCH 03/10] fixup! [libcxx][docs] Make test name pattern
documentation more obvious
(Fully) centralize all naming documentation in the rst-based documentation.
---
libcxx/docs/TestingLibcxx.rst | 41 ++++++++++++++++++++---------------
1 file changed, 23 insertions(+), 18 deletions(-)
diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst
index 5783cc6a21922c..b93f61388cda6c 100644
--- a/libcxx/docs/TestingLibcxx.rst
+++ b/libcxx/docs/TestingLibcxx.rst
@@ -334,11 +334,12 @@ Test names
----------
The names of test files have meaning for the libcxx-specific configuration of
-Lit. A complete description can be found in the docstring for the function
-``CxxStandardLibraryTest`` in the file ``libcxx/utils/libcxx/test/format.py``.
-
-For quick reference, here is an overview of some of the most useful patterns:
+Lit. Based on the pattern that matches the name of a test file, Lit will test
+the code contained therein in different ways. Refer to the `Lit Meaning of libcxx
+Test Filenames`_ when
+determining the names for new test files.
+.. _Lit Meaning of libcxx Test Filenames:
.. list-table:: Lit Meaning of libcxx Test Filenames
:widths: 25 75
:header-rows: 1
@@ -346,32 +347,36 @@ For quick reference, here is an overview of some of the most useful patterns:
* - Name Pattern
- Meaning
* - ``FOO.pass.cpp``
- - Compiles, links and runs successfully
+ - Checks whether the C++ code in the file compiles, links and runs successfully.
* - ``FOO.pass.mm``
- - Same as ``FOO.pass.cpp``, but for Objective-C++
+ - Same as ``FOO.pass.cpp``, but for Objective-C++.
* - ``FOO.compile.pass.cpp``
- - Compiles successfully, link and run not attempted
+ - Checks whether the C++ code in the file compiles successfully.
* - ``FOO.compile.pass.mm``
- - Same as ``FOO.compile.pass.cpp``, but for Objective-C++
+ - Same as ``FOO.compile.pass.cpp``, but for Objective-C++.
* - ``FOO.compile.fail.cpp``
- - Does not compile successfully
+ - Checks that the code in the file does *not* compile successfully.
+
+ * - ``FOO.verify.cpp``
+ - Compiles with clang-verify. This type of test is automatically marked as UNSUPPORTED if the compiler does not support clang-verify.
* - ``FOO.link.pass.cpp``
- - Compiles and links successfully, run not attempted
+ - Checks that the C++ code in the file compiles and links successfully -- no run attempted.
* - ``FOO.link.pass.mm``
- - Same as ``FOO.link.pass.cpp``, but for Objective-C++
+ - Same as ``FOO.link.pass.cpp``, but for Objective-C++.
* - ``FOO.link.fail.cpp``
- - Compiles successfully, but fails to link
+ - Checks whether the C++ code in the file fails to link after successful compilation.
+ * - ``FOO.link.fail.mm``
+ - Same as ``FOO.link.fail.cpp``, but for Objective-C++.
* - ``FOO.sh.<anything>``
- - A builtin Lit Shell test
-
+ - A *builtin Lit Shell* test.
* - ``FOO.gen.<anything>``
- - A ``.sh`` test that generates one or more Lit tests on the fly. Executing this test must generate one or more files as expected by LLVM split-file, and each generated file leads to a separate Lit test that runs that file as defined by the test format. This can be used to generate multiple Lit tests from a single source file, which is useful for testing repetitive properties in the library. Be careful not to abuse this since this is not a replacement for usual code reuse techniques.
-
- * - ``FOO.verify.cpp``
- - Compiles with clang-verify. This type of test is automatically marked as UNSUPPORTED if the compiler does not support clang-verify.
+ - A *builtin Lit Shell* test that generates one or more Lit tests on the fly. Executing this test must generate one or more files as expected
+ by LLVM split-file. Each generated file will drive an invocation of a separate Lit test. The format of the generated file will determine the type
+ of Lit test to be executed. This can be used to generate multiple Lit tests from a single source file, which is useful for testing repetitive properties
+ in the library. Be careful not to abuse this since this is not a replacement for usual code reuse techniques.
Benchmarks
==========
>From bb0d04bdb733308d8f45ccdbb7f974d1d36ce976 Mon Sep 17 00:00:00 2001
From: Will Hawkins <hawkinsw at obs.cr>
Date: Mon, 27 Nov 2023 09:19:29 -0500
Subject: [PATCH 04/10] fixup! [libcxx][docs] Make test name pattern
documentation more obvious
Point only to www documentation for test names (per @mordante's excellent feedback)
---
libcxx/utils/libcxx/test/format.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/libcxx/utils/libcxx/test/format.py b/libcxx/utils/libcxx/test/format.py
index ace208c98459de..19577f84d24ebe 100644
--- a/libcxx/utils/libcxx/test/format.py
+++ b/libcxx/utils/libcxx/test/format.py
@@ -153,7 +153,7 @@ class CxxStandardLibraryTest(lit.formats.FileBasedTest):
Lit tests are contained in files that follow a certain pattern. That
pattern determines the semantics of the test. See
- libcxx/docs/TestingLibcxx.rst or https://libcxx.llvm.org/TestingLibcxx.html
+ https://libcxx.llvm.org/TestingLibcxx.html#test-names
for a complete description of those semantics.
Substitution requirements
>From a6ab5ace1b72651ccb284f1567694adca282de6a Mon Sep 17 00:00:00 2001
From: Will Hawkins <whh8b at obs.cr>
Date: Wed, 13 Dec 2023 22:12:52 -0500
Subject: [PATCH 05/10] Update libcxx/docs/TestingLibcxx.rst
Be more clear about what type of shell we get when running these tests.
Co-authored-by: Louis Dionne <ldionne.2 at gmail.com>
---
libcxx/docs/TestingLibcxx.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst
index b93f61388cda6c..5ccac803f793cb 100644
--- a/libcxx/docs/TestingLibcxx.rst
+++ b/libcxx/docs/TestingLibcxx.rst
@@ -373,7 +373,7 @@ determining the names for new test files.
* - ``FOO.sh.<anything>``
- A *builtin Lit Shell* test.
* - ``FOO.gen.<anything>``
- - A *builtin Lit Shell* test that generates one or more Lit tests on the fly. Executing this test must generate one or more files as expected
+ - A variant of a *Lit Shell* test that generates one or more Lit tests on the fly. Executing this test must generate one or more files as expected
by LLVM split-file. Each generated file will drive an invocation of a separate Lit test. The format of the generated file will determine the type
of Lit test to be executed. This can be used to generate multiple Lit tests from a single source file, which is useful for testing repetitive properties
in the library. Be careful not to abuse this since this is not a replacement for usual code reuse techniques.
>From 1d77fc4fb56fd4d8d77223aab184a2aeac97abb2 Mon Sep 17 00:00:00 2001
From: Will Hawkins <whh8b at obs.cr>
Date: Wed, 13 Dec 2023 22:14:56 -0500
Subject: [PATCH 06/10] Update libcxx/utils/libcxx/test/format.py
Make it clearer how the tests are generated by libcxx to drive lit according to the patterns embedded in the filenames that contain tests.
Co-authored-by: Louis Dionne <ldionne.2 at gmail.com>
---
libcxx/utils/libcxx/test/format.py | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/libcxx/utils/libcxx/test/format.py b/libcxx/utils/libcxx/test/format.py
index 19577f84d24ebe..0a1f8716cd3966 100644
--- a/libcxx/utils/libcxx/test/format.py
+++ b/libcxx/utils/libcxx/test/format.py
@@ -151,8 +151,9 @@ class CxxStandardLibraryTest(lit.formats.FileBasedTest):
"""
Lit test format for the C++ Standard Library conformance test suite.
- Lit tests are contained in files that follow a certain pattern. That
- pattern determines the semantics of the test. See
+ Lit tests are contained in files that follow a certain pattern, which determines the semantics of the test.
+ Under the hood, we basically generate a builtin Lit shell test that follows the ShTest format, and perform
+ the appropriate operations (compile/link/run). See
https://libcxx.llvm.org/TestingLibcxx.html#test-names
for a complete description of those semantics.
>From 2cfdd279ed47fd4763845527fe48e455539ace16 Mon Sep 17 00:00:00 2001
From: Will Hawkins <hawkinsw at obs.cr>
Date: Wed, 13 Dec 2023 22:59:59 -0500
Subject: [PATCH 07/10] fixup! [libcxx][docs] Make test name pattern
documentation more obvious
Add documentation about ADDITIONAL_COMPILE_FLAGS and FILE_DEPENDENCIES.
---
libcxx/docs/TestingLibcxx.rst | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst
index 5ccac803f793cb..1b956434cefa6b 100644
--- a/libcxx/docs/TestingLibcxx.rst
+++ b/libcxx/docs/TestingLibcxx.rst
@@ -378,6 +378,36 @@ determining the names for new test files.
of Lit test to be executed. This can be used to generate multiple Lit tests from a single source file, which is useful for testing repetitive properties
in the library. Be careful not to abuse this since this is not a replacement for usual code reuse techniques.
+
+libc++-Specific Lit Features
+----------------------------
+
+Custom Directives
+~~~~~~~~~~~~~~~~~
+
+Lit has many directives built in (e.g., ``DEFINE``, ``UNSUPPORTED``). In addition to those directives, libcxx adds two additional libc++-specific directives that makes
+writing tests easier. See `libc++-specific Lit Directives`_ for more information about the ``FILE_DEPENDENCIES`` and ``ADDITIONAL_COMPILE_FLAGS`` libc++-specific directives.
+
+.. _libc++-specific Lit Directives:
+.. list-table:: libc++-specific Lit Directives
+ :widths: 20 35 45
+ :header-rows: 1
+
+ * - Directive
+ - Parameters
+ - Usage
+ * - ``FILE_DEPENDENCIES``
+ - ``// FILE_DEPENDENCIES: file, directory, /path/to/file, ...``
+ - The paths given to the ``FILE_DEPENDENCIES`` directive can specify directories or specific files upon which a given test depend. Copies of the files individually specified and
+ *all* the files in any specified directories will be placed in the *current working directory* of the test when it executes. All dependency directories and files
+ specified using relative paths will be anchored to the directory specified by the ``%S`` substitution (i.e., the source directory of the test being executed.).
+ * - ``ADDITIONAL_COMPILE_FLAGS``
+ - ``// ADDITIONAL_COMPILE_FLAGS: flag1, flag2, ...``
+ - The additional compiler flags specified using the ``ADDITIONAL_COMPILE_FLAGS`` libc++-specific Lit directive will be added to the invocation of ``clang`` used to build
+ the test executables. ``ADDITIONAL_COMPILE_FLAGS`` are only used with tests that must be built from source using clang. In other words, ``FOO.sh.<anything>`` tests will not use
+ the value of this directive. The clang compiler flags specified in the ``ADDITIONAL_COMPILE_FLAGS`` libc++-specific Lit directive are added *after* the compiler flags
+ specified in the ``%{compile_flags}`` substitution.
+
Benchmarks
==========
>From eaa17a23caf7103255d20762a1f8851ff36ac35c Mon Sep 17 00:00:00 2001
From: Will Hawkins <hawkinsw at obs.cr>
Date: Thu, 21 Dec 2023 15:57:19 -0500
Subject: [PATCH 08/10] fixup! [libcxx][docs] Make test name pattern
documentation more obvious
Address @ldionne's astute feedback.
---
libcxx/docs/TestingLibcxx.rst | 33 +++++++++++++++---------------
libcxx/utils/libcxx/test/format.py | 24 ----------------------
2 files changed, 17 insertions(+), 40 deletions(-)
diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst
index 1b956434cefa6b..3e9dab05814003 100644
--- a/libcxx/docs/TestingLibcxx.rst
+++ b/libcxx/docs/TestingLibcxx.rst
@@ -325,7 +325,7 @@ This macro is in a different header as ``assert_macros.h`` since it pulls in
additional headers.
.. note: This macro can only be used in test using C++20 or newer. The macro
- was added at a time where most of lib++'s C++17 support was complete.
+ was added at a time where most of libc++'s C++17 support was complete.
Since it is not expected to add this to existing tests no effort was
taken to make it work in earlier language versions.
@@ -333,14 +333,13 @@ additional headers.
Test names
----------
-The names of test files have meaning for the libcxx-specific configuration of
+The names of test files have meaning for the libc++-specific configuration of
Lit. Based on the pattern that matches the name of a test file, Lit will test
-the code contained therein in different ways. Refer to the `Lit Meaning of libcxx
-Test Filenames`_ when
-determining the names for new test files.
+the code contained therein in different ways. Refer to the `Lit Meaning of libc++
+Test Filenames`_ when determining the names for new test files.
-.. _Lit Meaning of libcxx Test Filenames:
-.. list-table:: Lit Meaning of libcxx Test Filenames
+.. _Lit Meaning of libc++ Test Filenames:
+.. list-table:: Lit Meaning of libc++ Test Filenames
:widths: 25 75
:header-rows: 1
@@ -385,7 +384,7 @@ libc++-Specific Lit Features
Custom Directives
~~~~~~~~~~~~~~~~~
-Lit has many directives built in (e.g., ``DEFINE``, ``UNSUPPORTED``). In addition to those directives, libcxx adds two additional libc++-specific directives that makes
+Lit has many directives built in (e.g., ``DEFINE``, ``UNSUPPORTED``). In addition to those directives, libc++ adds two additional libc++-specific directives that makes
writing tests easier. See `libc++-specific Lit Directives`_ for more information about the ``FILE_DEPENDENCIES`` and ``ADDITIONAL_COMPILE_FLAGS`` libc++-specific directives.
.. _libc++-specific Lit Directives:
@@ -398,15 +397,17 @@ writing tests easier. See `libc++-specific Lit Directives`_ for more information
- Usage
* - ``FILE_DEPENDENCIES``
- ``// FILE_DEPENDENCIES: file, directory, /path/to/file, ...``
- - The paths given to the ``FILE_DEPENDENCIES`` directive can specify directories or specific files upon which a given test depend. Copies of the files individually specified and
- *all* the files in any specified directories will be placed in the *current working directory* of the test when it executes. All dependency directories and files
- specified using relative paths will be anchored to the directory specified by the ``%S`` substitution (i.e., the source directory of the test being executed.).
+ - The paths given to the ``FILE_DEPENDENCIES`` directive can specify directories or specific files upon which a given test depend. For example, a test that requires some test
+ input stored in a data file would use this libc++-specific Lit directive. When a test file contains the ``FILE_DEPENDENCIES`` directive, Lit will collect the named files and copy
+ them to the directory represented by the ``%T`` substitution before the test executes. The copy is performed from the directory represented by the ``%S`` substitution
+ (i.e. the source directory of the test being executed) which makes it possible to use relative paths to specify the location of dependency files. After Lit copies
+ all the dependent files to the directory specified by the ``%T`` substitution, that directory should contain *all* the necessary inputs to run. In other words,
+ it should be possible to copy the contents of the directory specified by the ``%T`` substitution to a remote host where the execution of the test will actually occur.
* - ``ADDITIONAL_COMPILE_FLAGS``
- - ``// ADDITIONAL_COMPILE_FLAGS: flag1, flag2, ...``
- - The additional compiler flags specified using the ``ADDITIONAL_COMPILE_FLAGS`` libc++-specific Lit directive will be added to the invocation of ``clang`` used to build
- the test executables. ``ADDITIONAL_COMPILE_FLAGS`` are only used with tests that must be built from source using clang. In other words, ``FOO.sh.<anything>`` tests will not use
- the value of this directive. The clang compiler flags specified in the ``ADDITIONAL_COMPILE_FLAGS`` libc++-specific Lit directive are added *after* the compiler flags
- specified in the ``%{compile_flags}`` substitution.
+ - ``// ADDITIONAL_COMPILE_FLAGS: flag1 flag2 ...``
+ - The additional compiler flags specified by a space-separated list to the ``ADDITIONAL_COMPILE_FLAGS`` libc++-specific Lit directive will be added to the end of the ``%{compile_flags}``
+ substitution for the test that contains it. This libc++-specific Lit directive makes it possible to add special compilation flags without having to resort to writing a ``.sh.cpp`` test (see
+ `Lit Meaning of libc++ Test Filenames`_), more powerful but perhaps overkill.
Benchmarks
==========
diff --git a/libcxx/utils/libcxx/test/format.py b/libcxx/utils/libcxx/test/format.py
index 0a1f8716cd3966..5d84711bf5d282 100644
--- a/libcxx/utils/libcxx/test/format.py
+++ b/libcxx/utils/libcxx/test/format.py
@@ -173,30 +173,6 @@ class CxxStandardLibraryTest(lit.formats.FileBasedTest):
in the same command line. In other words, the test format doesn't perform
separate compilation and linking steps in this case.
-
- Additional supported directives
- ===============================
- In addition to everything that's supported in Lit ShTests, this test format
- also understands the following directives inside test files:
-
- // FILE_DEPENDENCIES: file, directory, /path/to/file
-
- This directive expresses that the test requires the provided files
- or directories in order to run. An example is a test that requires
- some test input stored in a data file. When a test file contains
- such a directive, this test format will collect them and copy them
- to the directory represented by %T. The intent is that %T contains
- all the inputs necessary to run the test, such that e.g. execution
- on a remote host can be done by simply copying %T to the host.
-
- // ADDITIONAL_COMPILE_FLAGS: flag1 flag2 flag3
-
- This directive will cause the provided flags to be added to the
- %{compile_flags} substitution for the test that contains it. This
- allows adding special compilation flags without having to use a
- .sh.cpp test, which would be more powerful but perhaps overkill.
-
-
Additional provided substitutions and features
==============================================
The test format will define the following substitutions for use inside tests:
>From 30ef8ed75099673dbd31e747734fc3b49d8aca17 Mon Sep 17 00:00:00 2001
From: Will Hawkins <hawkinsw at obs.cr>
Date: Tue, 26 Dec 2023 11:36:21 -0500
Subject: [PATCH 09/10] fixup! [libcxx][docs] Make test name pattern
documentation more obvious
Add link to Internals Manual for writing clang-verify tests.
---
libcxx/docs/TestingLibcxx.rst | 1 +
1 file changed, 1 insertion(+)
diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst
index 3e9dab05814003..4f27dd1f5ad249 100644
--- a/libcxx/docs/TestingLibcxx.rst
+++ b/libcxx/docs/TestingLibcxx.rst
@@ -359,6 +359,7 @@ Test Filenames`_ when determining the names for new test files.
* - ``FOO.verify.cpp``
- Compiles with clang-verify. This type of test is automatically marked as UNSUPPORTED if the compiler does not support clang-verify.
+ For additional information about how to write tests using ``clang-verify``, see the `Internals Manual <https://clang.llvm.org/docs/InternalsManual.html#verifying-diagnostics>`_.
* - ``FOO.link.pass.cpp``
- Checks that the C++ code in the file compiles and links successfully -- no run attempted.
>From b044d256facdb544b6a577b12ace74d57f1672a7 Mon Sep 17 00:00:00 2001
From: Will Hawkins <hawkinsw at obs.cr>
Date: Fri, 29 Dec 2023 20:16:38 -0500
Subject: [PATCH 10/10] fixup! [libcxx][docs] Make test name pattern
documentation more obvious
Add recommendation on when to write verify tests.
---
libcxx/docs/TestingLibcxx.rst | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/libcxx/docs/TestingLibcxx.rst b/libcxx/docs/TestingLibcxx.rst
index 4f27dd1f5ad249..2a4a26c493d671 100644
--- a/libcxx/docs/TestingLibcxx.rst
+++ b/libcxx/docs/TestingLibcxx.rst
@@ -351,7 +351,8 @@ Test Filenames`_ when determining the names for new test files.
- Same as ``FOO.pass.cpp``, but for Objective-C++.
* - ``FOO.compile.pass.cpp``
- - Checks whether the C++ code in the file compiles successfully.
+ - Checks whether the C++ code in the file compiles successfully. In general, prefer `compile` tests over `verify` tests,
+ subject to the specific recommendations, below, for when to write `verify` tests.
* - ``FOO.compile.pass.mm``
- Same as ``FOO.compile.pass.cpp``, but for Objective-C++.
* - ``FOO.compile.fail.cpp``
@@ -359,7 +360,14 @@ Test Filenames`_ when determining the names for new test files.
* - ``FOO.verify.cpp``
- Compiles with clang-verify. This type of test is automatically marked as UNSUPPORTED if the compiler does not support clang-verify.
- For additional information about how to write tests using ``clang-verify``, see the `Internals Manual <https://clang.llvm.org/docs/InternalsManual.html#verifying-diagnostics>`_.
+ For additional information about how to write `verify` tests, see the `Internals Manual <https://clang.llvm.org/docs/InternalsManual.html#verifying-diagnostics>`_.
+ Prefer `verify` tests over `compile` tests to test that compilation fails for a particular reason. For example, use a `verify` test
+ to ensure that
+
+ * an expected ``static_assert`` is triggered;
+ * the use of deprecated functions generates the proper warning;
+ * removed functions are no longer usable; or
+ * return values from functions marked ``[[nodiscard]]`` are stored.
* - ``FOO.link.pass.cpp``
- Checks that the C++ code in the file compiles and links successfully -- no run attempted.
More information about the libcxx-commits
mailing list