[PATCH][LNT] better code coverage in lit tests of lnt runtest
Chris Matthews
chris.matthews at apple.com
Tue Aug 13 15:59:40 PDT 2013
Thanks. This is one of my first lit patches, that all makes sense. Here is an updated version:
Index: tests/runtest/nt.py
===================================================================
--- tests/runtest/nt.py (revision 188243)
+++ tests/runtest/nt.py (working copy)
@@ -1,30 +1,74 @@
# Testing for the 'lnt runtest nt' module.
#
+# Check a basic nt run.
# RUN: lnt runtest nt \
# RUN: --sandbox %t.SANDBOX \
# RUN: --test-suite %S/Inputs/test-suite \
# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
# RUN: --no-timestamp > %t.log 2> %t.err
-#
# RUN: FileCheck --check-prefix CHECK-STDOUT < %t.log %s
-# RUN: FileCheck --check-prefix CHECK-STDERR < %t.err %s
+# RUN: FileCheck --check-prefix CHECK-BASIC < %t.err %s
+# RUN: FileCheck --check-prefix CHECK-REPORT < %t.SANDBOX/build/report.json %s
+# CHECK-REPORT: "run_order": "154331"
#
# CHECK-STDOUT: Import succeeded.
# CHECK-STDOUT: Added Machines: 1
# CHECK-STDOUT: Added Runs : 1
# CHECK-STDOUT: Added Tests : 130
# CHECK-STDOUT: --- Tested: 260 tests --
+#
+# CHECK-BASIC: inferred C++ compiler under test
+# CHECK-BASIC: checking source versions
+# CHECK-BASIC: using nickname
+# CHECK-BASIC: starting test
+# CHECK-BASIC: configuring
+# CHECK-BASIC: building test-suite tools
+# CHECK-BASIC: executing "nightly tests" with -j1
+# CHECK-BASIC: loading nightly test data
+# CHECK-BASIC: capturing machine information
+# CHECK-BASIC: generating report
+# CHECK-BASIC: submitting result to dummy instance
+#
+# Use the same sandbox again with --no-configure
+# RUN: lnt runtest nt \
+# RUN: --sandbox %t.SANDBOX \
+# RUN: --test-suite %S/Inputs/test-suite \
+# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
+# RUN: --no-timestamp --no-configure > %t.log 2> %t.err
+# RUN: FileCheck --check-prefix CHECK-NOCONF < %t.err %s
+# CHECK-NOCONF-NOT: configuring
+#
+# Manually set a run order.
+# RUN: lnt runtest nt \
+# RUN: --sandbox %t.SANDBOX \
+# RUN: --test-suite %S/Inputs/test-suite \
+# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
+# RUN: --no-timestamp --run-order=123 > %t.log 2> %t.err
+# RUN: FileCheck --check-prefix CHECK-RESULTS < %t.SANDBOX/build/report.json %s
+# CHECK-RESULTS: "run_order": "123"
+#
+# Change the machine name. Don't use LLVM.
+# RUN: lnt runtest nt \
+# RUN: --sandbox %t.SANDBOX \
+# RUN: --test-suite %S/Inputs/test-suite \
+# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
+# RUN: --no-auto-name foo > %t.log 2> %t.err
+# RUN: FileCheck --check-prefix CHECK-AUTONAME < %t.err %s
+# CHECK-AUTONAME: using nickname: 'foo'
+#
+# Run with sandboxing enabled.
+# RUN: lnt runtest nt \
+# RUN: --sandbox %t.SANDBOX \
+# RUN: --test-suite %S/Inputs/test-suite \
+# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
+# RUN: --no-timestamp --use-isolation > %t.log 2> %t.err
+# RUN: FileCheck --check-prefix CHECK-SANDBOX < %t.err %s
+#
+# CHECK-SANDBOX: creating sandbox profile
-# CHECK-STDERR: inferred C++ compiler under test
-# CHECK-STDERR: checking source versions
-# CHECK-STDERR: using nickname
-# CHECK-STDERR: starting test
-# CHECK-STDERR: configuring
-# CHECK-STDERR: building test-suite tools
-# CHECK-STDERR: executing "nightly tests" with -j1
-# CHECK-STDERR: loading nightly test data
-# CHECK-STDERR: capturing machine information
-# CHECK-STDERR: generating report
-# CHECK-STDERR: submitting result to dummy instance
-
-
+# Run without LLVM.
+# RUN: lnt runtest nt \
+# RUN: --sandbox %t.SANDBOX \
+# RUN: --test-suite %S/Inputs/test-suite \
+# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
+# RUN: --no-timestamp --without-llvm > %t.log 2> %t.err
Chris Matthews
chris.matthews at apple.com
On Aug 13, 2013, at 12:02 PM, Daniel Dunbar <daniel at zuster.org> wrote:
> Hi Chris,
>
> A couple comments:
>
> 1. It would be better to break down the FileCheck lines to be paired with each individual invocation (instead of collecting all of the output together in one log).
>
> 2. It would be good to "focus" the tests a bit more. For example, if one of the tests is going to check --no-auto-name, then the matching FileCheck test should verify that the right name is in the result, but not check a variety of other unassociated things. If those things need to be checked as well, better to write an explicit test to check all the generic stuff, and the focused stuff to check the various corners. This makes it easier to maintain the tests over time and more clear about what is actually trying to be tested with the different checks.
>
> - Daniel
>
>
>
> On Tue, Aug 13, 2013 at 11:35 AM, Chris Matthews <chris.matthews at apple.com> wrote:
> This patch adds better code coverage for the runtest lit tests by calling runtest with an assortment of options.
>
> Index: tests/runtest/nt.py
> ===================================================================
> --- tests/runtest/nt.py (revision 188243)
> +++ tests/runtest/nt.py (working copy)
> @@ -6,6 +6,34 @@
> # RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
> # RUN: --no-timestamp > %t.log 2> %t.err
> #
> +# Use the same sandbox again with --no-configure
> +# RUN: lnt runtest nt \
> +# RUN: --sandbox %t.SANDBOX \
> +# RUN: --test-suite %S/Inputs/test-suite \
> +# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
> +# RUN: --no-timestamp --no-configure >> %t.log 2>> %t.err
> +#
> +# Manually set a run order.
> +# RUN: lnt runtest nt \
> +# RUN: --sandbox %t.SANDBOX \
> +# RUN: --test-suite %S/Inputs/test-suite \
> +# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
> +# RUN: --no-timestamp --no-configure --run-order 123 >> %t.log 2>> %t.err
> +#
> +# Change the machine name. Don't use LLVM.
> +# RUN: lnt runtest nt \
> +# RUN: --sandbox %t.SANDBOX \
> +# RUN: --test-suite %S/Inputs/test-suite \
> +# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
> +# RUN: --without-llvm --no-auto-name foo >> %t.log 2>> %t.err
> +#
> +# Run with sandboxing enabled.
> +# RUN: lnt runtest nt \
> +# RUN: --sandbox %t.SANDBOX \
> +# RUN: --test-suite %S/Inputs/test-suite \
> +# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
> +# RUN: --no-timestamp --use-isolation >> %t.log 2>> %t.err
> +#
> # RUN: FileCheck --check-prefix CHECK-STDOUT < %t.log %s
> # RUN: FileCheck --check-prefix CHECK-STDERR < %t.err %s
> #
> @@ -26,5 +54,77 @@
> # CHECK-STDERR: capturing machine information
> # CHECK-STDERR: generating report
> # CHECK-STDERR: submitting result to dummy instance
> +# Testing for the 'lnt runtest nt' module.
> +#
> +#
> +# CHECK-STDOUT: Import succeeded.
> +# CHECK-STDOUT: Added Machines: 1
> +# CHECK-STDOUT: Added Runs : 1
> +# CHECK-STDOUT: Added Tests : 130
> +# CHECK-STDOUT: --- Tested: 260 tests --
>
> +# CHECK-STDERR: inferred C++ compiler under test
> +# CHECK-STDERR: checking source versions
> +# CHECK-STDERR: using nickname
> +# CHECK-STDERR: starting test
> +# CHECK-STDERR: building test-suite tools
> +# CHECK-STDERR: executing "nightly tests" with -j1
> +# CHECK-STDERR: loading nightly test data
> +# CHECK-STDERR: capturing machine information
> +# CHECK-STDERR: generating report
> +# CHECK-STDERR: submitting result to dummy instance
> +#
> +# CHECK-STDOUT: Import succeeded.
> +# CHECK-STDOUT: Added Machines: 1
> +# CHECK-STDOUT: Added Runs : 1
> +# CHECK-STDOUT: Added Tests : 130
> +# CHECK-STDOUT: --- Tested: 260 tests --
> +# CHECK-STDERR: inferred C++ compiler under test
> +# CHECK-STDERR: checking source versions
> +# CHECK-STDERR: using nickname
> +# CHECK-STDERR: starting test
> +# CHECK-STDERR: building test-suite tools
> +# CHECK-STDERR: executing "nightly tests" with -j1
> +# CHECK-STDERR: loading nightly test data
> +# CHECK-STDERR: capturing machine information
> +# CHECK-STDERR: generating report
> +# CHECK-STDERR: submitting result to dummy instance
> +#
> +# CHECK-STDOUT: Import succeeded.
> +# CHECK-STDOUT: Added Machines: 1
> +# CHECK-STDOUT: Added Runs : 1
> +# CHECK-STDOUT: Added Tests : 130
> +# CHECK-STDOUT: --- Tested: 260 tests --
> +# CHECK-STDERR: inferred C++ compiler under test
> +# CHECK-STDERR: checking source versions
> +# CHECK-STDERR: using nickname: 'foo'
> +# CHECK-STDERR: starting test
> +# CHECK-STDERR: configuring
> +# CHECK-STDERR: building test-suite tools
> +# CHECK-STDERR: executing "nightly tests" with -j1
> +# CHECK-STDERR: loading nightly test data
> +# CHECK-STDERR: capturing machine information
> +# CHECK-STDERR: generating report
> +# CHECK-STDERR: submitting result to dummy instance
> +#
> +# CHECK-STDOUT: Import succeeded.
> +# CHECK-STDOUT: Added Machines: 1
> +# CHECK-STDOUT: Added Runs : 1
> +# CHECK-STDOUT: Added Tests : 130
> +# CHECK-STDOUT: --- Tested: 260 tests --
>
> +# CHECK-STDERR: inferred C++ compiler under test
> +# CHECK-STDERR: checking source versions
> +# CHECK-STDERR: using nickname
> +# CHECK-STDERR: starting test
> +# CHECK-STDERR: configuring
> +# CHECK-STDERR: building test-suite tools
> +# CHECK-STDERR: creating sandbox profile
> +# CHECK-STDERR: executing "nightly tests" with -j1
> +# CHECK-STDERR: loading nightly test data
> +# CHECK-STDERR: capturing machine information
> +# CHECK-STDERR: generating report
> +# CHECK-STDERR: submitting result to dummy instance
> +# Testing for the 'lnt runtest nt' module.
>
>
>
> Chris Matthews
> chris.matthews at apple.com
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20130813/96e6d52f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: runtest_coverage.patch
Type: application/octet-stream
Size: 3591 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20130813/96e6d52f/attachment.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20130813/96e6d52f/attachment-0001.html>
More information about the llvm-commits
mailing list