[libcxx-commits] [libcxxabi] 80a2ddf - [libc++] Add an alternative Lit test format
Louis Dionne via libcxx-commits
libcxx-commits at lists.llvm.org
Fri Apr 3 08:36:25 PDT 2020
Author: Louis Dionne
Date: 2020-04-03T11:35:27-04:00
New Revision: 80a2ddf65ccd9837a7cdd0dfb96bb57863dd57d5
URL: https://github.com/llvm/llvm-project/commit/80a2ddf65ccd9837a7cdd0dfb96bb57863dd57d5
DIFF: https://github.com/llvm/llvm-project/commit/80a2ddf65ccd9837a7cdd0dfb96bb57863dd57d5.diff
LOG: [libc++] Add an alternative Lit test format
This new test format is simpler and more flexible. It creates Lit ShTests
on the fly that reuse existing substitutions (like %{cxx}) instead of
having complex logic in Python to run the tests. This has the benefit
that virtually no coding is required to customize how the test suite is
run -- one can achieve pretty much anything by defining the appropriate
substitutions in a simple lit.cfg file.
For example, in order to run the tests on an embedded device after
building with a specific SDK, one can set the %{cxx} and %{compile_flags}
substitutions to use that SDK, and the %{exec} substitution to the ssh.py
script currently used for .sh.cpp tests with a remote executor. Dealing with
the SSHExecutor becomes unnecessary, since all tests are treated like ShTests.
As a side effect of this design, configuration files for the test
suite can be as simple as:
config.substitutions.append(('%{cxx}', '<path-to-compiler>'))
config.substitutions.append(('%{compile_flags}', '<flags>'))
config.substitutions.append(('%{link_flags}', '<flags>'))
config.substitutions.append(('%{exec}', '<script-to-execute>'))
This should allow storing lit.cfg files for various configurations
directly in the repository instead of relying on complicated logic
in config.py to set up the right flags. I've found numerous problems
in that logic in the past years, and it seems like having simple and
explicit configuration files for the configurations we support is
going to solve most of these problems. Specifically, I am hoping to
store configuration files for testing other Standard Libraries in
the repository.
Improving the interaction with the test suite configuration is still a
work in progress, so for now this test format reuses the substitutions and
available features that are set up by the current config.py.
This new test format should support pretty much everything that the current
test format supports, however it will not be enabled by default at first to
make sure we're satisfied with it. For a short period of time, the new format
will require `--param=use_new_format=True` to be enabled, however it is a very
short term goal to replace the current testing format entirely and to simplify
the configuration accordingly.
Differential Revision: https://reviews.llvm.org/D77338
Added:
libcxx/test/libcxx/selftest/newformat/additional_compile_flags/substitutes-in-compile-flags.sh.cpp
libcxx/test/libcxx/selftest/newformat/additional_compile_flags/substitutes-in-run.sh.cpp
libcxx/test/libcxx/selftest/newformat/compile.fail.cpp/compile-error.compile.fail.cpp
libcxx/test/libcxx/selftest/newformat/compile.fail.cpp/compile-success.compile.fail.cpp
libcxx/test/libcxx/selftest/newformat/compile.fail.cpp/wrong-expected.compile.fail.cpp
libcxx/test/libcxx/selftest/newformat/compile.pass.cpp/compile-error.compile.pass.cpp
libcxx/test/libcxx/selftest/newformat/compile.pass.cpp/compile-success.compile.pass.cpp
libcxx/test/libcxx/selftest/newformat/compile.pass.cpp/link-error.compile.pass.cpp
libcxx/test/libcxx/selftest/newformat/compile.pass.cpp/run-error.compile.pass.cpp
libcxx/test/libcxx/selftest/newformat/fail.cpp/compile-error-with-verify.fail.cpp
libcxx/test/libcxx/selftest/newformat/fail.cpp/compile-error-without-verify.fail.cpp
libcxx/test/libcxx/selftest/newformat/fail.cpp/compile-success.fail.cpp
libcxx/test/libcxx/selftest/newformat/fail.cpp/wrong-expected.fail.cpp
libcxx/test/libcxx/selftest/newformat/fail.mm/compile-error-with-verify.fail.mm
libcxx/test/libcxx/selftest/newformat/fail.mm/compile-error-without-verify.fail.mm
libcxx/test/libcxx/selftest/newformat/fail.mm/compile-success.fail.mm
libcxx/test/libcxx/selftest/newformat/fail.mm/wrong-expected.fail.cpp
libcxx/test/libcxx/selftest/newformat/file_dependencies/no-dependencies.sh.cpp
libcxx/test/libcxx/selftest/newformat/file_dependencies/substitute-in-dependencies.sh.cpp
libcxx/test/libcxx/selftest/newformat/file_dependencies/substitute-in-run.sh.cpp
libcxx/test/libcxx/selftest/newformat/link.fail.cpp/compile-error.link.fail.cpp
libcxx/test/libcxx/selftest/newformat/link.fail.cpp/link-error.link.fail.cpp
libcxx/test/libcxx/selftest/newformat/link.fail.cpp/link-success.link.fail.cpp
libcxx/test/libcxx/selftest/newformat/link.pass.cpp/compile-error.link.pass.cpp
libcxx/test/libcxx/selftest/newformat/link.pass.cpp/link-error.link.pass.cpp
libcxx/test/libcxx/selftest/newformat/link.pass.cpp/link-success.link.pass.cpp
libcxx/test/libcxx/selftest/newformat/link.pass.cpp/run-error.link.pass.cpp
libcxx/test/libcxx/selftest/newformat/lit.local.cfg
libcxx/test/libcxx/selftest/newformat/pass.cpp/compile-error.pass.cpp
libcxx/test/libcxx/selftest/newformat/pass.cpp/link-error.pass.cpp
libcxx/test/libcxx/selftest/newformat/pass.cpp/run-error.pass.cpp
libcxx/test/libcxx/selftest/newformat/pass.cpp/run-success.pass.cpp
libcxx/test/libcxx/selftest/newformat/pass.mm/compile-error.pass.mm
libcxx/test/libcxx/selftest/newformat/pass.mm/link-error.pass.mm
libcxx/test/libcxx/selftest/newformat/pass.mm/no-arc.pass.mm
libcxx/test/libcxx/selftest/newformat/pass.mm/run-error.pass.mm
libcxx/test/libcxx/selftest/newformat/pass.mm/run-success.pass.mm
libcxx/test/libcxx/selftest/newformat/pass.mm/use-objective-cxx.pass.mm
libcxx/test/libcxx/selftest/newformat/run.fail.cpp/compile-error.run.fail.cpp
libcxx/test/libcxx/selftest/newformat/run.fail.cpp/link-error.run.fail.cpp
libcxx/test/libcxx/selftest/newformat/run.fail.cpp/run-error.run.fail.cpp
libcxx/test/libcxx/selftest/newformat/run.fail.cpp/run-success.run.fail.cpp
libcxx/test/libcxx/selftest/newformat/sh.cpp/empty.sh.cpp
libcxx/test/libcxx/selftest/newformat/sh.cpp/remote-substitutions.sh.cpp
libcxx/test/libcxx/selftest/newformat/sh.cpp/run-error.sh.cpp
libcxx/test/libcxx/selftest/newformat/sh.cpp/run-success.sh.cpp
libcxx/test/libcxx/selftest/newformat/sh.cpp/substitutions.sh.cpp
libcxx/utils/libcxx/test/newformat.py
Modified:
libcxx/test/libcxx/selftest/test.pass.mm
libcxx/test/lit.cfg
libcxxabi/test/lit.cfg
Removed:
################################################################################
diff --git a/libcxx/test/libcxx/selftest/newformat/additional_compile_flags/substitutes-in-compile-flags.sh.cpp b/libcxx/test/libcxx/selftest/newformat/additional_compile_flags/substitutes-in-compile-flags.sh.cpp
new file mode 100644
index 000000000000..6fa65a417524
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/additional_compile_flags/substitutes-in-compile-flags.sh.cpp
@@ -0,0 +1,13 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure that substitutions are performed inside additional compiler flags.
+
+// ADDITIONAL_COMPILE_FLAGS: -I %t.1
+// ADDITIONAL_COMPILE_FLAGS: -isystem %t.2 , -isysroot %t.3
+// RUN: echo "%{compile_flags}" | grep -e '-I %t.1 -isystem %t.2 -isysroot %t.3'
diff --git a/libcxx/test/libcxx/selftest/newformat/additional_compile_flags/substitutes-in-run.sh.cpp b/libcxx/test/libcxx/selftest/newformat/additional_compile_flags/substitutes-in-run.sh.cpp
new file mode 100644
index 000000000000..387b1a1c3440
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/additional_compile_flags/substitutes-in-run.sh.cpp
@@ -0,0 +1,15 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure that additional compiler flags are added to the %{compile_flags}
+// substitution.
+
+// ADDITIONAL_COMPILE_FLAGS: -foo
+// ADDITIONAL_COMPILE_FLAGS: -bar
+// ADDITIONAL_COMPILE_FLAGS: -baz, -foom
+// RUN: echo "%{compile_flags}" | grep -e '-foo -bar -baz -foom'
diff --git a/libcxx/test/libcxx/selftest/newformat/compile.fail.cpp/compile-error.compile.fail.cpp b/libcxx/test/libcxx/selftest/newformat/compile.fail.cpp/compile-error.compile.fail.cpp
new file mode 100644
index 000000000000..e54b8572a3ff
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/compile.fail.cpp/compile-error.compile.fail.cpp
@@ -0,0 +1,16 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: verify-support
+
+// Make sure the test passes if it fails at compile-time
+
+struct Foo { };
+typedef Foo::x x; // expected-error{{no type named 'x' in 'Foo'}}
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/compile.fail.cpp/compile-success.compile.fail.cpp b/libcxx/test/libcxx/selftest/newformat/compile.fail.cpp/compile-success.compile.fail.cpp
new file mode 100644
index 000000000000..1cec8a3632c1
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/compile.fail.cpp/compile-success.compile.fail.cpp
@@ -0,0 +1,13 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it succeeds at compile-time
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/compile.fail.cpp/wrong-expected.compile.fail.cpp b/libcxx/test/libcxx/selftest/newformat/compile.fail.cpp/wrong-expected.compile.fail.cpp
new file mode 100644
index 000000000000..8b02098e4c73
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/compile.fail.cpp/wrong-expected.compile.fail.cpp
@@ -0,0 +1,17 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails at compile-time, but the
+// expected-error is wrong.
+
+struct Foo { };
+typedef Foo::x x; // expected-error{{this is not found in the errors}}
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/compile.pass.cpp/compile-error.compile.pass.cpp b/libcxx/test/libcxx/selftest/newformat/compile.pass.cpp/compile-error.compile.pass.cpp
new file mode 100644
index 000000000000..9ba8a5edbed9
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/compile.pass.cpp/compile-error.compile.pass.cpp
@@ -0,0 +1,16 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails at compile-time
+
+struct Foo { };
+typedef Foo::x x;
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/compile.pass.cpp/compile-success.compile.pass.cpp b/libcxx/test/libcxx/selftest/newformat/compile.pass.cpp/compile-success.compile.pass.cpp
new file mode 100644
index 000000000000..36d46904c8fc
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/compile.pass.cpp/compile-success.compile.pass.cpp
@@ -0,0 +1,11 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure the test passes if it succeeds to compile
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/compile.pass.cpp/link-error.compile.pass.cpp b/libcxx/test/libcxx/selftest/newformat/compile.pass.cpp/link-error.compile.pass.cpp
new file mode 100644
index 000000000000..8dd6a5968828
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/compile.pass.cpp/link-error.compile.pass.cpp
@@ -0,0 +1,15 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure the test passes even if there's a link error, i.e. it isn't linked.
+
+extern void this_is_an_undefined_symbol();
+
+int main() {
+ this_is_an_undefined_symbol();
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/compile.pass.cpp/run-error.compile.pass.cpp b/libcxx/test/libcxx/selftest/newformat/compile.pass.cpp/run-error.compile.pass.cpp
new file mode 100644
index 000000000000..f3fbc64a38dd
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/compile.pass.cpp/run-error.compile.pass.cpp
@@ -0,0 +1,13 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure the test passes even if there's a runtime error, i.e. it isn't run.
+
+int main() {
+ return 1;
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/fail.cpp/compile-error-with-verify.fail.cpp b/libcxx/test/libcxx/selftest/newformat/fail.cpp/compile-error-with-verify.fail.cpp
new file mode 100644
index 000000000000..4442c784e12b
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/fail.cpp/compile-error-with-verify.fail.cpp
@@ -0,0 +1,14 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure the test passes if it fails at compile-time, with verify
+
+struct Foo { };
+typedef Foo::x x; // expected-error{{no type named 'x' in 'Foo'}}
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/fail.cpp/compile-error-without-verify.fail.cpp b/libcxx/test/libcxx/selftest/newformat/fail.cpp/compile-error-without-verify.fail.cpp
new file mode 100644
index 000000000000..ae8bf0b5e981
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/fail.cpp/compile-error-without-verify.fail.cpp
@@ -0,0 +1,14 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure the test passes if it fails at compile-time, without verify
+
+struct Foo { };
+typedef Foo::x x;
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/fail.cpp/compile-success.fail.cpp b/libcxx/test/libcxx/selftest/newformat/fail.cpp/compile-success.fail.cpp
new file mode 100644
index 000000000000..1cec8a3632c1
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/fail.cpp/compile-success.fail.cpp
@@ -0,0 +1,13 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it succeeds at compile-time
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/fail.cpp/wrong-expected.fail.cpp b/libcxx/test/libcxx/selftest/newformat/fail.cpp/wrong-expected.fail.cpp
new file mode 100644
index 000000000000..d06759fb8172
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/fail.cpp/wrong-expected.fail.cpp
@@ -0,0 +1,19 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: verify-support
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails at compile-time, but the
+// expected-error is wrong.
+
+struct Foo { };
+typedef Foo::x x; // expected-error{{this is not found in the errors}}
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/fail.mm/compile-error-with-verify.fail.mm b/libcxx/test/libcxx/selftest/newformat/fail.mm/compile-error-with-verify.fail.mm
new file mode 100644
index 000000000000..a45f05e3a925
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/fail.mm/compile-error-with-verify.fail.mm
@@ -0,0 +1,16 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// Make sure the test passes if it fails at compile-time, with verify
+
+struct Foo { };
+typedef Foo::x x; // expected-error{{no type named 'x' in 'Foo'}}
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/fail.mm/compile-error-without-verify.fail.mm b/libcxx/test/libcxx/selftest/newformat/fail.mm/compile-error-without-verify.fail.mm
new file mode 100644
index 000000000000..57edb18c7073
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/fail.mm/compile-error-without-verify.fail.mm
@@ -0,0 +1,16 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// Make sure the test passes if it fails at compile-time, without verify
+
+struct Foo { };
+typedef Foo::x x;
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/fail.mm/compile-success.fail.mm b/libcxx/test/libcxx/selftest/newformat/fail.mm/compile-success.fail.mm
new file mode 100644
index 000000000000..a802d9cb1252
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/fail.mm/compile-success.fail.mm
@@ -0,0 +1,15 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it succeeds at compile-time
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/fail.mm/wrong-expected.fail.cpp b/libcxx/test/libcxx/selftest/newformat/fail.mm/wrong-expected.fail.cpp
new file mode 100644
index 000000000000..d0f0f9344e37
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/fail.mm/wrong-expected.fail.cpp
@@ -0,0 +1,19 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails at compile-time, but the
+// expected-error is wrong.
+
+struct Foo { };
+typedef Foo::x x; // expected-error{{this is not found in the errors}}
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/file_dependencies/no-dependencies.sh.cpp b/libcxx/test/libcxx/selftest/newformat/file_dependencies/no-dependencies.sh.cpp
new file mode 100644
index 000000000000..adfb15ad50dd
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/file_dependencies/no-dependencies.sh.cpp
@@ -0,0 +1,13 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure that %{file_dependencies} is empty when no FILE_DEPENDENCIES
+// line appears. Amongst other things, this makes sure that we don't share
+// file dependencies across unrelated Lit tests.
+
+// RUN: test -z "%{file_dependencies}"
diff --git a/libcxx/test/libcxx/selftest/newformat/file_dependencies/substitute-in-dependencies.sh.cpp b/libcxx/test/libcxx/selftest/newformat/file_dependencies/substitute-in-dependencies.sh.cpp
new file mode 100644
index 000000000000..291afcb333c0
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/file_dependencies/substitute-in-dependencies.sh.cpp
@@ -0,0 +1,12 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure that lit substitutions are expanded inside FILE_DEPENDENCIES lines.
+
+// FILE_DEPENDENCIES: %s
+// RUN: echo %{file_dependencies} | grep 'substitute-in-dependencies.sh.cpp'
diff --git a/libcxx/test/libcxx/selftest/newformat/file_dependencies/substitute-in-run.sh.cpp b/libcxx/test/libcxx/selftest/newformat/file_dependencies/substitute-in-run.sh.cpp
new file mode 100644
index 000000000000..5bd0e335cd29
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/file_dependencies/substitute-in-run.sh.cpp
@@ -0,0 +1,17 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure %{file_dependencies} are substituted properly into RUN commands.
+
+// FILE_DEPENDENCIES: a.txt
+// FILE_DEPENDENCIES: b.txt, c.txt
+// FILE_DEPENDENCIES: /absolute/d.txt
+// RUN: echo %{file_dependencies} | grep 'a.txt'
+// RUN: echo %{file_dependencies} | grep 'b.txt'
+// RUN: echo %{file_dependencies} | grep 'c.txt'
+// RUN: echo %{file_dependencies} | grep '/absolute/d.txt'
diff --git a/libcxx/test/libcxx/selftest/newformat/link.fail.cpp/compile-error.link.fail.cpp b/libcxx/test/libcxx/selftest/newformat/link.fail.cpp/compile-error.link.fail.cpp
new file mode 100644
index 000000000000..b0d709a082d2
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/link.fail.cpp/compile-error.link.fail.cpp
@@ -0,0 +1,16 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails at compile-time.
+
+struct Foo { };
+typedef Foo::x x;
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/link.fail.cpp/link-error.link.fail.cpp b/libcxx/test/libcxx/selftest/newformat/link.fail.cpp/link-error.link.fail.cpp
new file mode 100644
index 000000000000..e40b6bf8f0ee
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/link.fail.cpp/link-error.link.fail.cpp
@@ -0,0 +1,15 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure the test passes if it fails at link time.
+
+extern void this_is_an_undefined_symbol();
+
+int main() {
+ this_is_an_undefined_symbol();
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/link.fail.cpp/link-success.link.fail.cpp b/libcxx/test/libcxx/selftest/newformat/link.fail.cpp/link-success.link.fail.cpp
new file mode 100644
index 000000000000..43d3cc3ee714
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/link.fail.cpp/link-success.link.fail.cpp
@@ -0,0 +1,13 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it succeeds to link.
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/link.pass.cpp/compile-error.link.pass.cpp b/libcxx/test/libcxx/selftest/newformat/link.pass.cpp/compile-error.link.pass.cpp
new file mode 100644
index 000000000000..b0d709a082d2
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/link.pass.cpp/compile-error.link.pass.cpp
@@ -0,0 +1,16 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails at compile-time.
+
+struct Foo { };
+typedef Foo::x x;
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/link.pass.cpp/link-error.link.pass.cpp b/libcxx/test/libcxx/selftest/newformat/link.pass.cpp/link-error.link.pass.cpp
new file mode 100644
index 000000000000..ed766675af01
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/link.pass.cpp/link-error.link.pass.cpp
@@ -0,0 +1,17 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails to link.
+
+extern void this_is_an_undefined_symbol();
+
+int main() {
+ this_is_an_undefined_symbol();
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/link.pass.cpp/link-success.link.pass.cpp b/libcxx/test/libcxx/selftest/newformat/link.pass.cpp/link-success.link.pass.cpp
new file mode 100644
index 000000000000..2187fbdeb98c
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/link.pass.cpp/link-success.link.pass.cpp
@@ -0,0 +1,11 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure the test passes if it succeeds to link.
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/link.pass.cpp/run-error.link.pass.cpp b/libcxx/test/libcxx/selftest/newformat/link.pass.cpp/run-error.link.pass.cpp
new file mode 100644
index 000000000000..04460a4994e0
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/link.pass.cpp/run-error.link.pass.cpp
@@ -0,0 +1,14 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure the test passes if it succeeds to link, even though it would have
+// failed at runtime.
+
+int main() {
+ return 1;
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/lit.local.cfg b/libcxx/test/libcxx/selftest/newformat/lit.local.cfg
new file mode 100644
index 000000000000..8789ca4253af
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/lit.local.cfg
@@ -0,0 +1,4 @@
+import libcxx.test.newformat
+
+if not isinstance(config.test_format, libcxx.test.newformat.CxxStandardLibraryTest):
+ config.unsupported = True
diff --git a/libcxx/test/libcxx/selftest/newformat/pass.cpp/compile-error.pass.cpp b/libcxx/test/libcxx/selftest/newformat/pass.cpp/compile-error.pass.cpp
new file mode 100644
index 000000000000..9ba8a5edbed9
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/pass.cpp/compile-error.pass.cpp
@@ -0,0 +1,16 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails at compile-time
+
+struct Foo { };
+typedef Foo::x x;
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/pass.cpp/link-error.pass.cpp b/libcxx/test/libcxx/selftest/newformat/pass.cpp/link-error.pass.cpp
new file mode 100644
index 000000000000..f62e23493331
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/pass.cpp/link-error.pass.cpp
@@ -0,0 +1,17 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails at link-time
+
+extern void this_is_an_undefined_symbol();
+
+int main() {
+ this_is_an_undefined_symbol();
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/pass.cpp/run-error.pass.cpp b/libcxx/test/libcxx/selftest/newformat/pass.cpp/run-error.pass.cpp
new file mode 100644
index 000000000000..999ee65e1e1e
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/pass.cpp/run-error.pass.cpp
@@ -0,0 +1,15 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails at runtime.
+
+int main() {
+ return 1;
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/pass.cpp/run-success.pass.cpp b/libcxx/test/libcxx/selftest/newformat/pass.cpp/run-success.pass.cpp
new file mode 100644
index 000000000000..85cf334bfaef
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/pass.cpp/run-success.pass.cpp
@@ -0,0 +1,13 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure the test passes pass if it succeeds at runtime.
+
+int main() {
+ return 0;
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/pass.mm/compile-error.pass.mm b/libcxx/test/libcxx/selftest/newformat/pass.mm/compile-error.pass.mm
new file mode 100644
index 000000000000..95c661887cad
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/pass.mm/compile-error.pass.mm
@@ -0,0 +1,18 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails at compile-time
+
+struct Foo { };
+typedef Foo::x x;
+
+int main() { }
diff --git a/libcxx/test/libcxx/selftest/newformat/pass.mm/link-error.pass.mm b/libcxx/test/libcxx/selftest/newformat/pass.mm/link-error.pass.mm
new file mode 100644
index 000000000000..5a33a2c5b5d1
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/pass.mm/link-error.pass.mm
@@ -0,0 +1,19 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails at link-time
+
+extern void this_is_an_undefined_symbol();
+
+int main() {
+ this_is_an_undefined_symbol();
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/pass.mm/no-arc.pass.mm b/libcxx/test/libcxx/selftest/newformat/pass.mm/no-arc.pass.mm
new file mode 100644
index 000000000000..cc1bc96469ba
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/pass.mm/no-arc.pass.mm
@@ -0,0 +1,19 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// Make sure ARC is not enabled by default in these tests.
+
+#if __has_feature(objc_arc)
+# error "arc should not be enabled by default"
+#endif
+
+int main() {
+ return 0;
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/pass.mm/run-error.pass.mm b/libcxx/test/libcxx/selftest/newformat/pass.mm/run-error.pass.mm
new file mode 100644
index 000000000000..03e640d332e1
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/pass.mm/run-error.pass.mm
@@ -0,0 +1,17 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails at runtime.
+
+int main() {
+ return 1;
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/pass.mm/run-success.pass.mm b/libcxx/test/libcxx/selftest/newformat/pass.mm/run-success.pass.mm
new file mode 100644
index 000000000000..471af21c2dc2
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/pass.mm/run-success.pass.mm
@@ -0,0 +1,15 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// Make sure the test passes pass if it succeeds at runtime.
+
+int main() {
+ return 0;
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/pass.mm/use-objective-cxx.pass.mm b/libcxx/test/libcxx/selftest/newformat/pass.mm/use-objective-cxx.pass.mm
new file mode 100644
index 000000000000..f45d3c1186b5
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/pass.mm/use-objective-cxx.pass.mm
@@ -0,0 +1,18 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: objective-c++
+
+// Make sure we can use Objective C++ features in these tests.
+
+ at interface Foo
+ at end
+
+int main() {
+ return 0;
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/run.fail.cpp/compile-error.run.fail.cpp b/libcxx/test/libcxx/selftest/newformat/run.fail.cpp/compile-error.run.fail.cpp
new file mode 100644
index 000000000000..94cff9c9854f
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/run.fail.cpp/compile-error.run.fail.cpp
@@ -0,0 +1,18 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails at compile-time
+
+struct Foo { };
+typedef Foo::x x;
+
+int main() {
+ return 1;
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/run.fail.cpp/link-error.run.fail.cpp b/libcxx/test/libcxx/selftest/newformat/run.fail.cpp/link-error.run.fail.cpp
new file mode 100644
index 000000000000..31a054272d15
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/run.fail.cpp/link-error.run.fail.cpp
@@ -0,0 +1,18 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// XFAIL: *
+
+// Make sure that the test DOES NOT pass if it fails at link-time
+
+extern void this_is_an_undefined_symbol();
+
+int main() {
+ this_is_an_undefined_symbol();
+ return 1;
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/run.fail.cpp/run-error.run.fail.cpp b/libcxx/test/libcxx/selftest/newformat/run.fail.cpp/run-error.run.fail.cpp
new file mode 100644
index 000000000000..29c0ce9ad44b
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/run.fail.cpp/run-error.run.fail.cpp
@@ -0,0 +1,13 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure the test passes if it fails at runtime.
+
+int main() {
+ return 1;
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/run.fail.cpp/run-success.run.fail.cpp b/libcxx/test/libcxx/selftest/newformat/run.fail.cpp/run-success.run.fail.cpp
new file mode 100644
index 000000000000..ebc53227e2d8
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/run.fail.cpp/run-success.run.fail.cpp
@@ -0,0 +1,15 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it succeeds at runtime.
+
+int main() {
+ return 0;
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/sh.cpp/empty.sh.cpp b/libcxx/test/libcxx/selftest/newformat/sh.cpp/empty.sh.cpp
new file mode 100644
index 000000000000..de97543319bc
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/sh.cpp/empty.sh.cpp
@@ -0,0 +1,11 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it has no 'RUN:' steps
diff --git a/libcxx/test/libcxx/selftest/newformat/sh.cpp/remote-substitutions.sh.cpp b/libcxx/test/libcxx/selftest/newformat/sh.cpp/remote-substitutions.sh.cpp
new file mode 100644
index 000000000000..40e48c6f3ac7
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/sh.cpp/remote-substitutions.sh.cpp
@@ -0,0 +1,30 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure that test-executables can appear in RUN lines and be executed
+// effectively. This somewhat
diff icult-to-understand test checks that when
+// we run with a remote executor, test-executables are copied to the remote
+// host and their path is fixed up (directly in the command-line) to their
+// path on the remote host instead of the local host.
+//
+// We also check that the path of test-executables is replaced whether they
+// appear first in the command-line or not.
+
+// FILE_DEPENDENCIES: %t.exe
+// RUN: %{cxx} %{compile_flags} %{link_flags} -o %t.exe %s
+// RUN: %{exec} %t.exe 0
+// RUN: %{exec} ! %t.exe 1
+
+#include <cassert>
+#include <cstdlib>
+
+int main(int argc, char** argv) {
+ assert(argc == 2);
+ int ret = std::atoi(argv[1]);
+ return ret;
+}
diff --git a/libcxx/test/libcxx/selftest/newformat/sh.cpp/run-error.sh.cpp b/libcxx/test/libcxx/selftest/newformat/sh.cpp/run-error.sh.cpp
new file mode 100644
index 000000000000..ce5cd4f03349
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/sh.cpp/run-error.sh.cpp
@@ -0,0 +1,13 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// XFAIL: *
+
+// Make sure the test DOES NOT pass if it fails to run
+
+// RUN: exit 1
diff --git a/libcxx/test/libcxx/selftest/newformat/sh.cpp/run-success.sh.cpp b/libcxx/test/libcxx/selftest/newformat/sh.cpp/run-success.sh.cpp
new file mode 100644
index 000000000000..9f2d58236f1b
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/sh.cpp/run-success.sh.cpp
@@ -0,0 +1,11 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure the test passes if it succeeds to run
+
+// RUN: exit 0
diff --git a/libcxx/test/libcxx/selftest/newformat/sh.cpp/substitutions.sh.cpp b/libcxx/test/libcxx/selftest/newformat/sh.cpp/substitutions.sh.cpp
new file mode 100644
index 000000000000..45ff9643bc66
--- /dev/null
+++ b/libcxx/test/libcxx/selftest/newformat/sh.cpp/substitutions.sh.cpp
@@ -0,0 +1,28 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// Make sure we have access to the following substitutions at all times:
+// - %{cxx}
+// - %{compile_flags}
+// - %{link_flags}
+// - %{exec}
+
+// FILE_DEPENDENCIES: %t.exe
+// RUN: %{cxx} %{compile_flags} %{link_flags} -o %t.exe %s
+// RUN: %{exec} %t.exe "HELLO"
+
+#include <cassert>
+#include <string>
+
+int main(int argc, char** argv) {
+ assert(argc == 2);
+
+ std::string arg = argv[1];
+ assert(arg == "HELLO");
+ return 0;
+}
diff --git a/libcxx/test/libcxx/selftest/test.pass.mm b/libcxx/test/libcxx/selftest/test.pass.mm
index ba4d36bbe211..2475c31e1895 100644
--- a/libcxx/test/libcxx/selftest/test.pass.mm
+++ b/libcxx/test/libcxx/selftest/test.pass.mm
@@ -7,6 +7,8 @@
//
//===----------------------------------------------------------------------===//
+// REQUIRES: objective-c++
+
#if __has_feature(objc_arc)
#error "arc should *not* be enabled"
#endif
diff --git a/libcxx/test/lit.cfg b/libcxx/test/lit.cfg
index 8ee592b43650..8f42dd7c468b 100644
--- a/libcxx/test/lit.cfg
+++ b/libcxx/test/lit.cfg
@@ -52,4 +52,9 @@ config_module = __import__(config_module_name, fromlist=['Configuration'])
configuration = config_module.Configuration(lit_config, config)
configuration.configure()
configuration.print_config_info()
-config.test_format = configuration.get_test_format()
+if lit_config.params.get('use_new_format', False):
+ lit_config.note("Using the experimental libc++ testing format")
+ import libcxx.test.newformat
+ config.test_format = libcxx.test.newformat.CxxStandardLibraryTest()
+else:
+ config.test_format = configuration.get_test_format()
diff --git a/libcxx/utils/libcxx/test/newformat.py b/libcxx/utils/libcxx/test/newformat.py
new file mode 100644
index 000000000000..649259b2f87a
--- /dev/null
+++ b/libcxx/utils/libcxx/test/newformat.py
@@ -0,0 +1,243 @@
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+import lit
+import os
+import pipes
+
+class CxxStandardLibraryTest(lit.formats.TestFormat):
+ """
+ Lit test format for the C++ Standard Library conformance test suite.
+
+ This test format is based on top of the ShTest format -- it basically
+ creates a shell script performing the right operations (compile/link/run)
+ based on the extension of the test file it encounters. It supports files
+ with the following extensions:
+
+ FOO.pass.cpp - Compiles, links and runs successfully
+ FOO.pass.mm - Same as .pass.cpp, but for Objective-C++
+ FOO.run.fail.cpp - Compiles and links successfully, but fails at runtime
+ FOO.compile.pass.cpp - Compiles successfully, link and run not attempted
+ FOO.compile.fail.cpp - Does not compile successfully. These tests run with
+ clang-verify.
+ FOO.link.pass.cpp - Compiles and links successfully, run not attempted
+ FOO.link.fail.cpp - Compiles successfully, but fails to link
+ FOO.sh.cpp - A builtin lit Shell test
+
+ FOO.fail.cpp - Does not compile successfully -- run with clang-verify
+ if any expected-meow appears in the file, otherwise
+ just test that compilation fails. This is supported
+ only for backwards compatibility with the test suite
+ FOO.fail.mm - Same as .fail.cpp, but for Objective-C++
+
+ The test format operates by assuming that each test's configuration provides
+ the following substitutions, which it will reuse in the shell scripts it
+ constructs:
+ %{cxx} - A command that can be used to invoke the compiler
+ %{compile_flags} - Flags to use when compiling a test case
+ %{link_flags} - Flags to use when linking a test case
+ %{exec} - A command to prefix the execution of executables
+
+ Note that when building an executable (as opposed to only compiling a source
+ file), both %{compile_flags} and %{link_flags} will be used in the same
+ command line. In other words, the test format doesn't perform separate
+ compilation and linking steps in this case.
+
+
+ In addition to everything that's supported in Lit ShTests, this test format
+ also understands the following directives inside test files:
+
+ // FILE_DEPENDENCIES: file, directory, /path/to/file
+
+ This directive expresses that the test requires the provided files
+ or directories in order to run. An example is a test that requires
+ some test input stored in a data file. When a test file contains
+ such a directive, this test format will collect them and make them
+ available in a special %{file_dependencies} substitution. The intent
+ is that if one needs to e.g. execute tests on a remote host, the
+ %{exec} substitution could use %{file_dependencies} to know which
+ files and directories to copy to the remote host.
+
+ // ADDITIONAL_COMPILE_FLAGS: flag1, flag2, flag3
+
+ This directive will cause the provided flags to be added to the
+ %{compile_flags} substitution for the test that contains it. This
+ allows adding special compilation flags without having to use a
+ .sh.cpp test, which would be more powerful but perhaps overkill.
+
+
+ Design note:
+ This test format never implicitly disables a type of test. For example,
+ we could be tempted to automatically mark `.compile.fail.cpp` tests as
+ UNSUPPORTED when clang-verify isn't supported by the compiler. However,
+ this sort of logic has been known to cause tests to be ignored in the
+ past, so we favour having tests mark themselves as unsupported explicitly.
+
+ This test format still needs work in the following areas:
+ - It doesn't support the MODULES_DEFINES from the old test format, but
+ that can be achieved easily with ADDITIONAL_COMPILE_FLAGS.
+ - It is unknown how well it works on Windows yet.
+ """
+ def getTestsInDirectory(self, testSuite, pathInSuite, litConfig, localConfig):
+ SUPPORTED_SUFFIXES = ['.pass.cpp', '.run.fail.cpp',
+ '.pass.mm',
+ '.compile.pass.cpp', '.compile.fail.cpp',
+ '.link.pass.cpp', '.link.fail.cpp',
+ '.sh.cpp',
+ '.fail.cpp', '.fail.mm']
+ sourcePath = testSuite.getSourcePath(pathInSuite)
+ for filename in os.listdir(sourcePath):
+ # Ignore dot files and excluded tests.
+ if filename.startswith('.') or filename in localConfig.excludes:
+ continue
+
+ filepath = os.path.join(sourcePath, filename)
+ if not os.path.isdir(filepath):
+ if any([filename.endswith(ext) for ext in SUPPORTED_SUFFIXES]):
+ yield lit.Test.Test(testSuite, pathInSuite + (filename,), localConfig)
+
+ def _checkSubstitutions(self, substitutions):
+ substitutions = [s for (s, _) in substitutions]
+ for s in ['%{cxx}', '%{compile_flags}', '%{link_flags}', '%{exec}']:
+ assert s in substitutions, "Required substitution {} was not provided".format(s)
+
+ # Determine whether -verify should be used for a given test. We use -verify
+ # if the compiler supports it and there's at least one -verify tag in the
+ # source file.
+ #
+ # This is only supported for backwards compatibility with .fail.cpp tests.
+ def _useVerify(self, test, litConfig):
+ VERIFY_TAGS = (b'expected-note', b'expected-remark',
+ b'expected-warning', b'expected-error',
+ b'expected-no-diagnostics')
+ with open(test.getSourcePath(), 'rb') as f:
+ contents = f.read()
+ testContainsTags = any(tag in contents for tag in VERIFY_TAGS)
+
+ command = "echo | %{cxx} -xc++ - -Werror -fsyntax-only -Xclang -verify-ignore-unexpected"
+ result = lit.TestRunner.executeShTest(test, litConfig,
+ useExternalSh=True,
+ preamble_commands=[command])
+ compilerSupportsVerify = result.code != lit.Test.FAIL
+ return compilerSupportsVerify and testContainsTags
+
+ def execute(self, test, litConfig):
+ self._checkSubstitutions(test.config.substitutions)
+ VERIFY_FLAGS = '-Xclang -verify -Xclang -verify-ignore-unexpected=note -ferror-limit=0'
+ filename = test.path_in_suite[-1]
+ if filename.endswith('.sh.cpp'):
+ steps = [ ] # The steps are already in the script
+ return self._executeShTest(test, litConfig, steps)
+ elif filename.endswith('.compile.pass.cpp'):
+ steps = [
+ "%dbg(COMPILED WITH) %{cxx} %s %{compile_flags} -fsyntax-only"
+ ]
+ return self._executeShTest(test, litConfig, steps)
+ elif filename.endswith('.compile.fail.cpp'):
+ steps = [
+ "%dbg(COMPILED WITH) %{cxx} %s %{compile_flags} -fsyntax-only " + VERIFY_FLAGS
+ ]
+ return self._executeShTest(test, litConfig, steps)
+ elif filename.endswith('.link.pass.cpp'):
+ steps = [
+ "%dbg(COMPILED WITH) %{cxx} %s %{compile_flags} %{link_flags} -o %t.exe"
+ ]
+ return self._executeShTest(test, litConfig, steps)
+ elif filename.endswith('.link.fail.cpp'):
+ steps = [
+ "%dbg(COMPILED WITH) %{cxx} %s %{compile_flags} -c -o %t.o",
+ "%dbg(LINKED WITH) ! %{cxx} %t.o %{link_flags} -o %t.exe"
+ ]
+ return self._executeShTest(test, litConfig, steps)
+ elif filename.endswith('.run.fail.cpp'):
+ steps = [
+ "%dbg(COMPILED WITH) %{cxx} %s %{compile_flags} %{link_flags} -o %t.exe",
+ "%dbg(EXECUTED AS) %{exec} ! %t.exe"
+ ]
+ return self._executeShTest(test, litConfig, steps, fileDependencies=['%t.exe'])
+ # Make sure to check these ones last, since they will match other
+ # suffixes above too.
+ elif filename.endswith('.pass.cpp') or filename.endswith('.pass.mm'):
+ steps = [
+ "%dbg(COMPILED WITH) %{cxx} %s %{compile_flags} %{link_flags} -o %t.exe",
+ "%dbg(EXECUTED AS) %{exec} %t.exe"
+ ]
+ return self._executeShTest(test, litConfig, steps, fileDependencies=['%t.exe'])
+ # This is the same as .compile.fail.cpp, provided for backwards
+ # compatibility with the test suite.
+ elif filename.endswith('.fail.cpp') or filename.endswith('.fail.mm'):
+ if self._useVerify(test, litConfig):
+ steps = [
+ "%dbg(COMPILED WITH) %{cxx} %s %{compile_flags} -fsyntax-only " + VERIFY_FLAGS
+ ]
+ else:
+ steps = [
+ "%dbg(COMPILED WITH) ! %{cxx} %s %{compile_flags} -fsyntax-only"
+ ]
+ return self._executeShTest(test, litConfig, steps)
+ else:
+ return lit.Test.Result(lit.Test.FAIL, "Unknown test suffix for '{}'".format(filename))
+
+ # Utility function to add compile flags in lit.local.cfg files.
+ def addCompileFlags(self, config, *flags):
+ string = ' '.join(flags)
+ config.substitutions = [(s, x + ' ' + string) if s == '%{compile_flags}' else (s, x) for (s, x) in config.substitutions]
+
+ # Modified version of lit.TestRunner.executeShTest to handle custom parsers correctly.
+ def _executeShTest(self, test, litConfig, steps, fileDependencies=None):
+ recursiveExpansionLimit = 10 # TODO: Use the value in litConfig once we set it.
+ if test.config.unsupported:
+ return lit.Test.Result(lit.Test.UNSUPPORTED, 'Test is unsupported')
+
+ additionalCompileFlags = []
+ fileDependencies = fileDependencies or []
+ parsers = [
+ lit.TestRunner.IntegratedTestKeywordParser('FILE_DEPENDENCIES:',
+ lit.TestRunner.ParserKind.LIST,
+ initial_value=fileDependencies),
+ lit.TestRunner.IntegratedTestKeywordParser('ADDITIONAL_COMPILE_FLAGS:',
+ lit.TestRunner.ParserKind.LIST,
+ initial_value=additionalCompileFlags)
+ ]
+
+ script = list(steps)
+ parsed = lit.TestRunner.parseIntegratedTestScript(test, additional_parsers=parsers,
+ require_script=not script)
+ if isinstance(parsed, lit.Test.Result):
+ return parsed
+ script += parsed
+
+ if litConfig.noExecute:
+ return lit.Test.Result(lit.Test.PASS)
+
+ # Add compile flags specified with ADDITIONAL_COMPILE_FLAGS.
+ self.addCompileFlags(test.config, *additionalCompileFlags)
+
+ tmpDir, tmpBase = lit.TestRunner.getTempPaths(test)
+ useExternalSh = True
+ substitutions = lit.TestRunner.getDefaultSubstitutions(test, tmpDir, tmpBase,
+ normalize_slashes=useExternalSh)
+
+ # Perform substitutions inside FILE_DEPENDENCIES lines (or injected dependencies).
+ # This allows using variables like %t in file dependencies. Also note that we really
+ # need to resolve %{file_dependencies} now, because otherwise we won't be able to
+ # make all paths absolute below.
+ fileDependencies = lit.TestRunner.applySubstitutions(fileDependencies, substitutions,
+ recursion_limit=recursiveExpansionLimit)
+
+ # Add the %{file_dependencies} substitution before we perform substitutions
+ # inside the script.
+ testDir = os.path.dirname(test.getSourcePath())
+ fileDependencies = [f if os.path.isabs(f) else os.path.join(testDir, f) for f in fileDependencies]
+ substitutions.append(('%{file_dependencies}', ' '.join(map(pipes.quote, fileDependencies))))
+
+ # Perform substitution in the script itself.
+ script = lit.TestRunner.applySubstitutions(script, substitutions,
+ recursion_limit=recursiveExpansionLimit)
+
+ return lit.TestRunner._runShTest(test, litConfig, useExternalSh, script, tmpBase)
diff --git a/libcxxabi/test/lit.cfg b/libcxxabi/test/lit.cfg
index 2e23c564b792..5d79f169de78 100644
--- a/libcxxabi/test/lit.cfg
+++ b/libcxxabi/test/lit.cfg
@@ -64,4 +64,9 @@ config_module = __import__(config_module_name, fromlist=['Configuration'])
configuration = config_module.Configuration(lit_config, config)
configuration.configure()
configuration.print_config_info()
-config.test_format = configuration.get_test_format()
+if lit_config.params.get('use_new_format', False):
+ lit_config.note("Using the experimental libc++ testing format")
+ import libcxx.test.newformat
+ config.test_format = libcxx.test.newformat.CxxStandardLibraryTest()
+else:
+ config.test_format = configuration.get_test_format()
More information about the libcxx-commits
mailing list