[PATCH][LNT] better code coverage in lit tests of lnt runtest

[Daniel Dunbar]

Hi Chris,

A couple comments:

1. It would be better to break down the FileCheck lines to be paired with
each individual invocation (instead of collecting all of the output
together in one log).

2. It would be good to "focus" the tests a bit more. For example, if one of
the tests is going to check --no-auto-name, then the matching FileCheck
test should verify that the right name is in the result, but not check a
variety of other unassociated things. If those things need to be checked as
well, better to write an explicit test to check all the generic stuff, and
the focused stuff to check the various corners. This makes it easier to
maintain the tests over time and more clear about what is actually trying
to be tested with the different checks.

 - Daniel



On Tue, Aug 13, 2013 at 11:35 AM, Chris Matthews
<chris.matthews at apple.com>wrote:

> This patch adds better code coverage for the runtest lit tests by calling
> runtest with an assortment of options.
>
> Index: tests/runtest/nt.py
> ===================================================================
> --- tests/runtest/nt.py (revision 188243)
> +++ tests/runtest/nt.py (working copy)
> @@ -6,6 +6,34 @@
>  # RUN:   --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
>  # RUN:   --no-timestamp > %t.log 2> %t.err
>  #
> +# Use the same sandbox again with --no-configure
> +# RUN: lnt runtest nt \
> +# RUN:   --sandbox %t.SANDBOX \
> +# RUN:   --test-suite %S/Inputs/test-suite \
> +# RUN:   --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
> +# RUN:   --no-timestamp --no-configure >> %t.log 2>> %t.err
> +#
> +# Manually set a run order.
> +# RUN: lnt runtest nt \
> +# RUN:   --sandbox %t.SANDBOX \
> +# RUN:   --test-suite %S/Inputs/test-suite \
> +# RUN:   --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
> +# RUN:   --no-timestamp --no-configure --run-order 123 >> %t.log 2>>
> %t.err
> +#
> +# Change the machine name. Don't use LLVM.
> +# RUN: lnt runtest nt \
> +# RUN:   --sandbox %t.SANDBOX \
> +# RUN:   --test-suite %S/Inputs/test-suite \
> +# RUN:   --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
> +# RUN:   --without-llvm --no-auto-name foo >> %t.log 2>> %t.err
> +#
> +# Run with sandboxing enabled.
> +# RUN: lnt runtest nt \
> +# RUN:   --sandbox %t.SANDBOX \
> +# RUN:   --test-suite %S/Inputs/test-suite \
> +# RUN:   --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
> +# RUN:   --no-timestamp --use-isolation >> %t.log 2>> %t.err
> +#
>  # RUN: FileCheck --check-prefix CHECK-STDOUT < %t.log %s
>  # RUN: FileCheck --check-prefix CHECK-STDERR < %t.err %s
>  #
> @@ -26,5 +54,77 @@
>  # CHECK-STDERR: capturing machine information
>  # CHECK-STDERR: generating report
>  # CHECK-STDERR: submitting result to dummy instance
> +# Testing for the 'lnt runtest nt' module.
> +#
> +#
> +# CHECK-STDOUT: Import succeeded.
> +# CHECK-STDOUT: Added Machines: 1
> +# CHECK-STDOUT: Added Runs    : 1
> +# CHECK-STDOUT: Added Tests   : 130
> +# CHECK-STDOUT: --- Tested: 260 tests --
>
> +# CHECK-STDERR: inferred C++ compiler under test
> +# CHECK-STDERR: checking source versions
> +# CHECK-STDERR: using nickname
> +# CHECK-STDERR: starting test
> +# CHECK-STDERR: building test-suite tools
> +# CHECK-STDERR: executing "nightly tests" with -j1
> +# CHECK-STDERR: loading nightly test data
> +# CHECK-STDERR: capturing machine information
> +# CHECK-STDERR: generating report
> +# CHECK-STDERR: submitting result to dummy instance
> +#
> +# CHECK-STDOUT: Import succeeded.
> +# CHECK-STDOUT: Added Machines: 1
> +# CHECK-STDOUT: Added Runs    : 1
> +# CHECK-STDOUT: Added Tests   : 130
> +# CHECK-STDOUT: --- Tested: 260 tests --
> +# CHECK-STDERR: inferred C++ compiler under test
> +# CHECK-STDERR: checking source versions
> +# CHECK-STDERR: using nickname
> +# CHECK-STDERR: starting test
> +# CHECK-STDERR: building test-suite tools
> +# CHECK-STDERR: executing "nightly tests" with -j1
> +# CHECK-STDERR: loading nightly test data
> +# CHECK-STDERR: capturing machine information
> +# CHECK-STDERR: generating report
> +# CHECK-STDERR: submitting result to dummy instance
> +#
> +# CHECK-STDOUT: Import succeeded.
> +# CHECK-STDOUT: Added Machines: 1
> +# CHECK-STDOUT: Added Runs    : 1
> +# CHECK-STDOUT: Added Tests   : 130
> +# CHECK-STDOUT: --- Tested: 260 tests --
> +# CHECK-STDERR: inferred C++ compiler under test
> +# CHECK-STDERR: checking source versions
> +# CHECK-STDERR: using nickname: 'foo'
> +# CHECK-STDERR: starting test
> +# CHECK-STDERR: configuring
> +# CHECK-STDERR: building test-suite tools
> +# CHECK-STDERR: executing "nightly tests" with -j1
> +# CHECK-STDERR: loading nightly test data
> +# CHECK-STDERR: capturing machine information
> +# CHECK-STDERR: generating report
> +# CHECK-STDERR: submitting result to dummy instance
> +#
> +# CHECK-STDOUT: Import succeeded.
> +# CHECK-STDOUT: Added Machines: 1
> +# CHECK-STDOUT: Added Runs    : 1
> +# CHECK-STDOUT: Added Tests   : 130
> +# CHECK-STDOUT: --- Tested: 260 tests --
>
> +# CHECK-STDERR: inferred C++ compiler under test
> +# CHECK-STDERR: checking source versions
> +# CHECK-STDERR: using nickname
> +# CHECK-STDERR: starting test
> +# CHECK-STDERR: configuring
> +# CHECK-STDERR: building test-suite tools
> +# CHECK-STDERR: creating sandbox profile
> +# CHECK-STDERR: executing "nightly tests" with -j1
> +# CHECK-STDERR: loading nightly test data
> +# CHECK-STDERR: capturing machine information
> +# CHECK-STDERR: generating report
> +# CHECK-STDERR: submitting result to dummy instance
> +# Testing for the 'lnt runtest nt' module.
>
>
>
> Chris Matthews
> chris.matthews at apple.com
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20130813/8100c56c/attachment.html>