[PATCH][LNT] better code coverage in lit tests of lnt runtest

Daniel Dunbar daniel at zuster.org
Wed Aug 14 08:59:09 PDT 2013 

Looks great, thanks!

 - Daniel


On Tue, Aug 13, 2013 at 3:59 PM, Chris Matthews <chris.matthews at apple.com>wrote:

> Thanks. This is one of my first lit patches, that all makes sense.  Here
> is an updated version:
>
> Index: tests/runtest/nt.py
> ===================================================================
> --- tests/runtest/nt.py (revision 188243)
> +++ tests/runtest/nt.py (working copy)
> @@ -1,30 +1,74 @@
>  # Testing for the 'lnt runtest nt' module.
>  #
> +# Check a basic nt run.
>  # RUN: lnt runtest nt \
>  # RUN:   --sandbox %t.SANDBOX \
>  # RUN:   --test-suite %S/Inputs/test-suite \
>  # RUN:   --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
>  # RUN:   --no-timestamp > %t.log 2> %t.err
> -#
>  # RUN: FileCheck --check-prefix CHECK-STDOUT < %t.log %s
> -# RUN: FileCheck --check-prefix CHECK-STDERR < %t.err %s
> +# RUN: FileCheck --check-prefix CHECK-BASIC < %t.err %s
> +# RUN: FileCheck --check-prefix CHECK-REPORT <
> %t.SANDBOX/build/report.json %s
> +# CHECK-REPORT: "run_order": "154331"
>  #
>  # CHECK-STDOUT: Import succeeded.
>  # CHECK-STDOUT: Added Machines: 1
>  # CHECK-STDOUT: Added Runs    : 1
>  # CHECK-STDOUT: Added Tests   : 130
>  # CHECK-STDOUT: --- Tested: 260 tests --
> +#
> +# CHECK-BASIC: inferred C++ compiler under test
> +# CHECK-BASIC: checking source versions
> +# CHECK-BASIC: using nickname
> +# CHECK-BASIC: starting test
> +# CHECK-BASIC: configuring
> +# CHECK-BASIC: building test-suite tools
> +# CHECK-BASIC: executing "nightly tests" with -j1
> +# CHECK-BASIC: loading nightly test data
> +# CHECK-BASIC: capturing machine information
> +# CHECK-BASIC: generating report
> +# CHECK-BASIC: submitting result to dummy instance
> +#
> +# Use the same sandbox again with --no-configure
> +# RUN: lnt runtest nt \
> +# RUN:   --sandbox %t.SANDBOX \
> +# RUN:   --test-suite %S/Inputs/test-suite \
> +# RUN:   --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
> +# RUN:   --no-timestamp --no-configure > %t.log 2> %t.err
> +# RUN: FileCheck --check-prefix CHECK-NOCONF < %t.err %s
> +# CHECK-NOCONF-NOT: configuring
> +#
> +# Manually set a run order.
> +# RUN: lnt runtest nt \
> +# RUN:   --sandbox %t.SANDBOX \
> +# RUN:   --test-suite %S/Inputs/test-suite \
> +# RUN:   --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
> +# RUN:   --no-timestamp --run-order=123 > %t.log 2> %t.err
> +# RUN: FileCheck --check-prefix CHECK-RESULTS <
> %t.SANDBOX/build/report.json %s
> +# CHECK-RESULTS: "run_order": "123"
> +#
> +# Change the machine name. Don't use LLVM.
> +# RUN: lnt runtest nt \
> +# RUN:   --sandbox %t.SANDBOX \
> +# RUN:   --test-suite %S/Inputs/test-suite \
> +# RUN:   --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
> +# RUN:   --no-auto-name foo > %t.log 2> %t.err
> +# RUN: FileCheck --check-prefix CHECK-AUTONAME < %t.err %s
> +# CHECK-AUTONAME: using nickname: 'foo'
> +#
> +# Run with sandboxing enabled.
> +# RUN: lnt runtest nt \
> +# RUN:   --sandbox %t.SANDBOX \
> +# RUN:   --test-suite %S/Inputs/test-suite \
> +# RUN:   --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
> +# RUN:   --no-timestamp --use-isolation > %t.log 2> %t.err
> +# RUN: FileCheck --check-prefix CHECK-SANDBOX < %t.err %s
> +#
> +# CHECK-SANDBOX: creating sandbox profile
>
> -# CHECK-STDERR: inferred C++ compiler under test
> -# CHECK-STDERR: checking source versions
> -# CHECK-STDERR: using nickname
> -# CHECK-STDERR: starting test
> -# CHECK-STDERR: configuring
> -# CHECK-STDERR: building test-suite tools
> -# CHECK-STDERR: executing "nightly tests" with -j1
> -# CHECK-STDERR: loading nightly test data
> -# CHECK-STDERR: capturing machine information
> -# CHECK-STDERR: generating report
> -# CHECK-STDERR: submitting result to dummy instance
> -
> -
> +# Run without LLVM.
> +# RUN: lnt runtest nt \
> +# RUN:   --sandbox %t.SANDBOX \
> +# RUN:   --test-suite %S/Inputs/test-suite \
> +# RUN:   --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
> +# RUN:   --no-timestamp --without-llvm > %t.log 2> %t.err
>
>
>
> Chris Matthews
> chris.matthews at apple.com
>
> On Aug 13, 2013, at 12:02 PM, Daniel Dunbar <daniel at zuster.org> wrote:
>
> Hi Chris,
>
> A couple comments:
>
> 1. It would be better to break down the FileCheck lines to be paired with
> each individual invocation (instead of collecting all of the output
> together in one log).
>
> 2. It would be good to "focus" the tests a bit more. For example, if one
> of the tests is going to check --no-auto-name, then the matching FileCheck
> test should verify that the right name is in the result, but not check a
> variety of other unassociated things. If those things need to be checked as
> well, better to write an explicit test to check all the generic stuff, and
> the focused stuff to check the various corners. This makes it easier to
> maintain the tests over time and more clear about what is actually trying
> to be tested with the different checks.
>
>  - Daniel
>
>
>
> On Tue, Aug 13, 2013 at 11:35 AM, Chris Matthews <chris.matthews at apple.com
> > wrote:
>
>> This patch adds better code coverage for the runtest lit tests by calling
>> runtest with an assortment of options.
>>
>> Index: tests/runtest/nt.py
>> ===================================================================
>> --- tests/runtest/nt.py (revision 188243)
>> +++ tests/runtest/nt.py (working copy)
>> @@ -6,6 +6,34 @@
>>  # RUN:   --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
>>  # RUN:   --no-timestamp > %t.log 2> %t.err
>>  #
>> +# Use the same sandbox again with --no-configure
>> +# RUN: lnt runtest nt \
>> +# RUN:   --sandbox %t.SANDBOX \
>> +# RUN:   --test-suite %S/Inputs/test-suite \
>> +# RUN:   --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
>> +# RUN:   --no-timestamp --no-configure >> %t.log 2>> %t.err
>> +#
>> +# Manually set a run order.
>> +# RUN: lnt runtest nt \
>> +# RUN:   --sandbox %t.SANDBOX \
>> +# RUN:   --test-suite %S/Inputs/test-suite \
>> +# RUN:   --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
>> +# RUN:   --no-timestamp --no-configure --run-order 123 >> %t.log 2>>
>> %t.err
>> +#
>> +# Change the machine name. Don't use LLVM.
>> +# RUN: lnt runtest nt \
>> +# RUN:   --sandbox %t.SANDBOX \
>> +# RUN:   --test-suite %S/Inputs/test-suite \
>> +# RUN:   --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
>> +# RUN:   --without-llvm --no-auto-name foo >> %t.log 2>> %t.err
>> +#
>> +# Run with sandboxing enabled.
>> +# RUN: lnt runtest nt \
>> +# RUN:   --sandbox %t.SANDBOX \
>> +# RUN:   --test-suite %S/Inputs/test-suite \
>> +# RUN:   --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
>> +# RUN:   --no-timestamp --use-isolation >> %t.log 2>> %t.err
>> +#
>>  # RUN: FileCheck --check-prefix CHECK-STDOUT < %t.log %s
>>  # RUN: FileCheck --check-prefix CHECK-STDERR < %t.err %s
>>  #
>> @@ -26,5 +54,77 @@
>>  # CHECK-STDERR: capturing machine information
>>  # CHECK-STDERR: generating report
>>  # CHECK-STDERR: submitting result to dummy instance
>> +# Testing for the 'lnt runtest nt' module.
>> +#
>> +#
>> +# CHECK-STDOUT: Import succeeded.
>> +# CHECK-STDOUT: Added Machines: 1
>> +# CHECK-STDOUT: Added Runs    : 1
>> +# CHECK-STDOUT: Added Tests   : 130
>> +# CHECK-STDOUT: --- Tested: 260 tests --
>>
>> +# CHECK-STDERR: inferred C++ compiler under test
>> +# CHECK-STDERR: checking source versions
>> +# CHECK-STDERR: using nickname
>> +# CHECK-STDERR: starting test
>> +# CHECK-STDERR: building test-suite tools
>> +# CHECK-STDERR: executing "nightly tests" with -j1
>> +# CHECK-STDERR: loading nightly test data
>> +# CHECK-STDERR: capturing machine information
>> +# CHECK-STDERR: generating report
>> +# CHECK-STDERR: submitting result to dummy instance
>> +#
>> +# CHECK-STDOUT: Import succeeded.
>> +# CHECK-STDOUT: Added Machines: 1
>> +# CHECK-STDOUT: Added Runs    : 1
>> +# CHECK-STDOUT: Added Tests   : 130
>> +# CHECK-STDOUT: --- Tested: 260 tests --
>> +# CHECK-STDERR: inferred C++ compiler under test
>> +# CHECK-STDERR: checking source versions
>> +# CHECK-STDERR: using nickname
>> +# CHECK-STDERR: starting test
>> +# CHECK-STDERR: building test-suite tools
>> +# CHECK-STDERR: executing "nightly tests" with -j1
>> +# CHECK-STDERR: loading nightly test data
>> +# CHECK-STDERR: capturing machine information
>> +# CHECK-STDERR: generating report
>> +# CHECK-STDERR: submitting result to dummy instance
>> +#
>> +# CHECK-STDOUT: Import succeeded.
>> +# CHECK-STDOUT: Added Machines: 1
>> +# CHECK-STDOUT: Added Runs    : 1
>> +# CHECK-STDOUT: Added Tests   : 130
>> +# CHECK-STDOUT: --- Tested: 260 tests --
>> +# CHECK-STDERR: inferred C++ compiler under test
>> +# CHECK-STDERR: checking source versions
>> +# CHECK-STDERR: using nickname: 'foo'
>> +# CHECK-STDERR: starting test
>> +# CHECK-STDERR: configuring
>> +# CHECK-STDERR: building test-suite tools
>> +# CHECK-STDERR: executing "nightly tests" with -j1
>> +# CHECK-STDERR: loading nightly test data
>> +# CHECK-STDERR: capturing machine information
>> +# CHECK-STDERR: generating report
>> +# CHECK-STDERR: submitting result to dummy instance
>> +#
>> +# CHECK-STDOUT: Import succeeded.
>> +# CHECK-STDOUT: Added Machines: 1
>> +# CHECK-STDOUT: Added Runs    : 1
>> +# CHECK-STDOUT: Added Tests   : 130
>> +# CHECK-STDOUT: --- Tested: 260 tests --
>>
>> +# CHECK-STDERR: inferred C++ compiler under test
>> +# CHECK-STDERR: checking source versions
>> +# CHECK-STDERR: using nickname
>> +# CHECK-STDERR: starting test
>> +# CHECK-STDERR: configuring
>> +# CHECK-STDERR: building test-suite tools
>> +# CHECK-STDERR: creating sandbox profile
>> +# CHECK-STDERR: executing "nightly tests" with -j1
>> +# CHECK-STDERR: loading nightly test data
>> +# CHECK-STDERR: capturing machine information
>> +# CHECK-STDERR: generating report
>> +# CHECK-STDERR: submitting result to dummy instance
>> +# Testing for the 'lnt runtest nt' module.
>>
>>
>>
>> Chris Matthews
>> chris.matthews at apple.com
>>
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>>
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20130814/e6185cfc/attachment.html>

More information about the llvm-commits mailing list