<div dir="ltr">Looks great, thanks!<div><br></div><div> - Daniel</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Aug 13, 2013 at 3:59 PM, Chris Matthews <span dir="ltr"><<a href="mailto:chris.matthews@apple.com" target="_blank">chris.matthews@apple.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Thanks. This is one of my first lit patches, that all makes sense. Here is an updated version:<div>
<br></div><div><div class="im"><div>Index: tests/runtest/nt.py</div><div>===================================================================</div><div>--- tests/runtest/nt.py<span style="white-space:pre-wrap"> </span>(revision 188243)</div>
<div>+++ tests/runtest/nt.py<span style="white-space:pre-wrap"> </span>(working copy)</div></div><div>@@ -1,30 +1,74 @@</div><div class="im"><div> # Testing for the 'lnt runtest nt' module.</div></div><div> #</div>
<div>+# Check a basic nt run.</div><div> # RUN: lnt runtest nt \</div><div> # RUN: --sandbox %t.SANDBOX \</div><div> # RUN: --test-suite %S/Inputs/test-suite \</div><div class="im"><div> # RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \</div>
<div> # RUN: --no-timestamp > %t.log 2> %t.err</div></div><div class="im"><div>-#</div><div> # RUN: FileCheck --check-prefix CHECK-STDOUT < %t.log %s</div></div><div>-# RUN: FileCheck --check-prefix CHECK-STDERR < %t.err %s</div>
<div>+# RUN: FileCheck --check-prefix CHECK-BASIC < %t.err %s</div><div>+# RUN: FileCheck --check-prefix CHECK-REPORT < %t.SANDBOX/build/report.json %s</div><div>+# CHECK-REPORT: "run_order": "154331" </div>
<div> #</div><div> # CHECK-STDOUT: Import succeeded.</div><div> # CHECK-STDOUT: Added Machines: 1</div><div> # CHECK-STDOUT: Added Runs : 1</div><div> # CHECK-STDOUT: Added Tests : 130</div><div class="im"><div> # CHECK-STDOUT: --- Tested: 260 tests --</div>
<div>+#</div></div><div>+# CHECK-BASIC: inferred C++ compiler under test</div><div>+# CHECK-BASIC: checking source versions</div><div>+# CHECK-BASIC: using nickname</div><div>+# CHECK-BASIC: starting test</div><div>+# CHECK-BASIC: configuring</div>
<div>+# CHECK-BASIC: building test-suite tools</div><div>+# CHECK-BASIC: executing "nightly tests" with -j1</div><div>+# CHECK-BASIC: loading nightly test data</div><div>+# CHECK-BASIC: capturing machine information</div>
<div>+# CHECK-BASIC: generating report</div><div>+# CHECK-BASIC: submitting result to dummy instance</div><div>+#</div><div class="im"><div>+# Use the same sandbox again with --no-configure</div><div>+# RUN: lnt runtest nt \</div>
<div>+# RUN: --sandbox %t.SANDBOX \</div><div>+# RUN: --test-suite %S/Inputs/test-suite \</div><div>+# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \</div><div>+# RUN: --no-timestamp --no-configure > %t.log 2> %t.err</div>
</div><div>+# RUN: FileCheck --check-prefix CHECK-NOCONF < %t.err %s</div><div>+# CHECK-NOCONF-NOT: configuring</div><div class="im"><div>+#</div><div>+# Manually set a run order.</div><div>+# RUN: lnt runtest nt \</div>
<div>+# RUN: --sandbox %t.SANDBOX \</div><div>+# RUN: --test-suite %S/Inputs/test-suite \</div><div>+# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \</div></div><div>+# RUN: --no-timestamp --run-order=123 > %t.log 2> %t.err</div>
<div>+# RUN: FileCheck --check-prefix CHECK-RESULTS < %t.SANDBOX/build/report.json %s</div><div>+# CHECK-RESULTS: "run_order": "123" </div><div class="im"><div>+#</div><div>+# Change the machine name. Don't use LLVM.</div>
<div>+# RUN: lnt runtest nt \</div><div>+# RUN: --sandbox %t.SANDBOX \</div><div>+# RUN: --test-suite %S/Inputs/test-suite \</div><div>+# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \</div></div><div>+# RUN: --no-auto-name foo > %t.log 2> %t.err</div>
<div>+# RUN: FileCheck --check-prefix CHECK-AUTONAME < %t.err %s</div><div>+# CHECK-AUTONAME: using nickname: 'foo'</div><div class="im"><div>+#</div><div>+# Run with sandboxing enabled.</div><div>+# RUN: lnt runtest nt \</div>
<div>+# RUN: --sandbox %t.SANDBOX \</div><div>+# RUN: --test-suite %S/Inputs/test-suite \</div><div>+# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \</div><div>+# RUN: --no-timestamp --use-isolation > %t.log 2> %t.err</div>
</div><div>+# RUN: FileCheck --check-prefix CHECK-SANDBOX < %t.err %s</div><div>+#</div><div>+# CHECK-SANDBOX: creating sandbox profile</div><div class="im"><div> </div><div>-# CHECK-STDERR: inferred C++ compiler under test</div>
</div><div>-# CHECK-STDERR: checking source versions</div><div>-# CHECK-STDERR: using nickname</div><div>-# CHECK-STDERR: starting test</div><div>-# CHECK-STDERR: configuring</div><div class="im"><div>-# CHECK-STDERR: building test-suite tools</div>
<div>-# CHECK-STDERR: executing "nightly tests" with -j1</div><div>-# CHECK-STDERR: loading nightly test data</div></div><div>-# CHECK-STDERR: capturing machine information</div><div>-# CHECK-STDERR: generating report</div>
<div>-# CHECK-STDERR: submitting result to dummy instance</div><div>-</div><div>-</div><div>+# Run without LLVM.</div><div class="im"><div>+# RUN: lnt runtest nt \</div><div>+# RUN: --sandbox %t.SANDBOX \</div><div>+# RUN: --test-suite %S/Inputs/test-suite \</div>
<div>+# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \</div></div><div>+# RUN: --no-timestamp --without-llvm > %t.log 2> %t.err</div><div><br></div><div></div></div></div><br><div style="word-wrap:break-word">
<div><div></div><div>
<div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px"><div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px">
<div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px"><div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px">
<br><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;display:inline!important;font-weight:normal;float:none;line-height:normal;text-transform:none;white-space:normal;font-family:Helvetica;word-spacing:0px">Chris Matthews</span><br style="line-height:normal;text-indent:0px;letter-spacing:normal;text-align:start;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;display:inline!important;font-weight:normal;float:none;line-height:normal;text-transform:none;white-space:normal;font-family:Helvetica;word-spacing:0px"><a href="mailto:chris.matthews@apple.com" target="_blank">chris.matthews@apple.com</a></span><br style="line-height:normal;text-indent:0px;letter-spacing:normal;text-align:start;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">
</div></div></div></div>
</div>
<br><div><div>On Aug 13, 2013, at 12:02 PM, Daniel Dunbar <<a href="mailto:daniel@zuster.org" target="_blank">daniel@zuster.org</a>> wrote:</div><br><blockquote type="cite"><div dir="ltr">Hi Chris,<div><br></div><div>
A couple comments:</div><div><br></div><div>1. It would be better to break down the FileCheck lines to be paired with each individual invocation (instead of collecting all of the output together in one log).</div>
<div><br></div><div>2. It would be good to "focus" the tests a bit more. For example, if one of the tests is going to check --no-auto-name, then the matching FileCheck test should verify that the right name is in the result, but not check a variety of other unassociated things. If those things need to be checked as well, better to write an explicit test to check all the generic stuff, and the focused stuff to check the various corners. This makes it easier to maintain the tests over time and more clear about what is actually trying to be tested with the different checks.</div>
<div><br></div><div> - Daniel</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Aug 13, 2013 at 11:35 AM, Chris Matthews <span dir="ltr"><<a href="mailto:chris.matthews@apple.com" target="_blank">chris.matthews@apple.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">This patch adds better code coverage for the runtest lit tests by calling runtest with an assortment of options.<div>
<br></div><div><div>Index: tests/runtest/nt.py</div><div>===================================================================</div><div>--- tests/runtest/nt.py<span style="white-space:pre-wrap"> </span>(revision 188243)</div>
<div>+++ tests/runtest/nt.py<span style="white-space:pre-wrap"> </span>(working copy)</div><div>@@ -6,6 +6,34 @@</div><div> # RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \</div><div> # RUN: --no-timestamp > %t.log 2> %t.err</div>
<div> #</div><div>+# Use the same sandbox again with --no-configure</div><div>+# RUN: lnt runtest nt \</div><div>+# RUN: --sandbox %t.SANDBOX \</div><div>+# RUN: --test-suite %S/Inputs/test-suite \</div><div>+# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \</div>
<div>+# RUN: --no-timestamp --no-configure >> %t.log 2>> %t.err</div><div>+#</div><div>+# Manually set a run order.</div><div>+# RUN: lnt runtest nt \</div><div>+# RUN: --sandbox %t.SANDBOX \</div><div>+# RUN: --test-suite %S/Inputs/test-suite \</div>
<div>+# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \</div><div>+# RUN: --no-timestamp --no-configure --run-order 123 >> %t.log 2>> %t.err</div><div>+#</div><div>+# Change the machine name. Don't use LLVM.</div>
<div>+# RUN: lnt runtest nt \</div><div>+# RUN: --sandbox %t.SANDBOX \</div><div>+# RUN: --test-suite %S/Inputs/test-suite \</div><div>+# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \</div><div>+# RUN: --without-llvm --no-auto-name foo >> %t.log 2>> %t.err</div>
<div>+#</div><div>+# Run with sandboxing enabled.</div><div>+# RUN: lnt runtest nt \</div><div>+# RUN: --sandbox %t.SANDBOX \</div><div>+# RUN: --test-suite %S/Inputs/test-suite \</div><div>+# RUN: --cc %{shared_inputs}/FakeCompilers/clang-r154331 \</div>
<div>+# RUN: --no-timestamp --use-isolation >> %t.log 2>> %t.err</div><div>+#</div><div> # RUN: FileCheck --check-prefix CHECK-STDOUT < %t.log %s</div><div> # RUN: FileCheck --check-prefix CHECK-STDERR < %t.err %s</div>
<div> #</div><div>@@ -26,5 +54,77 @@</div><div> # CHECK-STDERR: capturing machine information</div><div> # CHECK-STDERR: generating report</div><div> # CHECK-STDERR: submitting result to dummy instance</div><div>+# Testing for the 'lnt runtest nt' module.</div>
<div>+#</div><div>+#</div><div>+# CHECK-STDOUT: Import succeeded.</div><div>+# CHECK-STDOUT: Added Machines: 1</div><div>+# CHECK-STDOUT: Added Runs : 1</div><div>+# CHECK-STDOUT: Added Tests : 130</div><div>+# CHECK-STDOUT: --- Tested: 260 tests --</div>
<div> </div><div>+# CHECK-STDERR: inferred C++ compiler under test</div><div>+# CHECK-STDERR: checking source versions</div><div>+# CHECK-STDERR: using nickname</div><div>+# CHECK-STDERR: starting test</div><div>+# CHECK-STDERR: building test-suite tools</div>
<div>+# CHECK-STDERR: executing "nightly tests" with -j1</div><div>+# CHECK-STDERR: loading nightly test data</div><div>+# CHECK-STDERR: capturing machine information</div><div>+# CHECK-STDERR: generating report</div>
<div>+# CHECK-STDERR: submitting result to dummy instance</div><div>+#</div><div>+# CHECK-STDOUT: Import succeeded.</div><div>+# CHECK-STDOUT: Added Machines: 1</div><div>+# CHECK-STDOUT: Added Runs : 1</div><div>+# CHECK-STDOUT: Added Tests : 130</div>
<div>+# CHECK-STDOUT: --- Tested: 260 tests --</div><div>+# CHECK-STDERR: inferred C++ compiler under test</div><div>+# CHECK-STDERR: checking source versions</div><div>+# CHECK-STDERR: using nickname</div><div>+# CHECK-STDERR: starting test</div>
<div>+# CHECK-STDERR: building test-suite tools</div><div>+# CHECK-STDERR: executing "nightly tests" with -j1</div><div>+# CHECK-STDERR: loading nightly test data</div><div>+# CHECK-STDERR: capturing machine information</div>
<div>+# CHECK-STDERR: generating report</div><div>+# CHECK-STDERR: submitting result to dummy instance</div><div>+#</div><div>+# CHECK-STDOUT: Import succeeded.</div><div>+# CHECK-STDOUT: Added Machines: 1</div><div>+# CHECK-STDOUT: Added Runs : 1</div>
<div>+# CHECK-STDOUT: Added Tests : 130</div><div>+# CHECK-STDOUT: --- Tested: 260 tests --</div><div>+# CHECK-STDERR: inferred C++ compiler under test</div><div>+# CHECK-STDERR: checking source versions</div><div>+# CHECK-STDERR: using nickname: 'foo'</div>
<div>+# CHECK-STDERR: starting test</div><div>+# CHECK-STDERR: configuring</div><div>+# CHECK-STDERR: building test-suite tools</div><div>+# CHECK-STDERR: executing "nightly tests" with -j1</div><div>+# CHECK-STDERR: loading nightly test data</div>
<div>+# CHECK-STDERR: capturing machine information</div><div>+# CHECK-STDERR: generating report</div><div>+# CHECK-STDERR: submitting result to dummy instance</div><div>+#</div><div>+# CHECK-STDOUT: Import succeeded.</div>
<div>+# CHECK-STDOUT: Added Machines: 1</div><div>+# CHECK-STDOUT: Added Runs : 1</div><div>+# CHECK-STDOUT: Added Tests : 130</div><div>+# CHECK-STDOUT: --- Tested: 260 tests --</div><div> </div><div>+# CHECK-STDERR: inferred C++ compiler under test</div>
<div>+# CHECK-STDERR: checking source versions</div><div>+# CHECK-STDERR: using nickname</div><div>+# CHECK-STDERR: starting test</div><div>+# CHECK-STDERR: configuring</div><div>+# CHECK-STDERR: building test-suite tools</div>
<div>+# CHECK-STDERR: creating sandbox profile</div><div>+# CHECK-STDERR: executing "nightly tests" with -j1</div><div>+# CHECK-STDERR: loading nightly test data</div><div>+# CHECK-STDERR: capturing machine information</div>
<div>+# CHECK-STDERR: generating report</div><div>+# CHECK-STDERR: submitting result to dummy instance</div><div>+# Testing for the 'lnt runtest nt' module.</div><div><br></div><div>
<div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px"><div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px">
<div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px"><div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px">
</div></div></div></div></div></div></div><br><div style="word-wrap:break-word"><div><div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px">
<div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px"><div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px">
<div style="text-indent:0px;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-wrap:break-word;word-spacing:0px"><br><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;display:inline!important;font-weight:normal;float:none;line-height:normal;text-transform:none;white-space:normal;font-family:Helvetica;word-spacing:0px">Chris Matthews</span><br style="line-height:normal;text-indent:0px;letter-spacing:normal;text-align:start;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;display:inline!important;font-weight:normal;float:none;line-height:normal;text-transform:none;white-space:normal;font-family:Helvetica;word-spacing:0px"><a href="mailto:chris.matthews@apple.com" target="_blank">chris.matthews@apple.com</a></span><br style="line-height:normal;text-indent:0px;letter-spacing:normal;text-align:start;font-variant:normal;text-transform:none;font-style:normal;white-space:normal;font-family:Helvetica;font-weight:normal;word-spacing:0px">
</div></div></div></div>
</div>
<br></div><br>_______________________________________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@cs.uiuc.edu" target="_blank">llvm-commits@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div></div><br></blockquote></div><br></div>