[LNT] r262224 - [cmake] Don't exit 1 if LIT exits 1

[James Molloy via llvm-commits]

Author: jamesm
Date: Mon Feb 29 08:27:19 2016
New Revision: 262224

URL: http://llvm.org/viewvc/llvm-project?rev=262224&view=rev
Log:
[cmake] Don't exit 1 if LIT exits 1

We expect test failures. LIT shouldn't be expected to exit with 0 all the time. We must ensure we don't bail out early and refuse to create the test failure report!

Modified:
    lnt/trunk/lnt/tests/test_suite.py
    lnt/trunk/tests/runtest/test_suite.py

Modified: lnt/trunk/lnt/tests/test_suite.py
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/tests/test_suite.py?rev=262224&r1=262223&r2=262224&view=diff
==============================================================================
--- lnt/trunk/lnt/tests/test_suite.py (original)
+++ lnt/trunk/lnt/tests/test_suite.py Mon Feb 29 08:27:19 2016
@@ -404,12 +404,16 @@ class TestSuiteTest(BuiltinTest):
             extra_args = ['--no-execute']
 
         note('Testing...')
-        self._check_call([lit_cmd,
-                          '-v',
-                          '-j', str(self._test_threads()),
-                          subdir,
-                          '-o', output_json_path.name] + extra_args)
-
+        try:
+            self._check_call([lit_cmd,
+                              '-v',
+                              '-j', str(self._test_threads()),
+                              subdir,
+                              '-o', output_json_path.name] + extra_args)
+        except subprocess.CalledProcessError:
+            # LIT is expected to exit with code 1 if there were test
+            # failures!
+            pass
         return json.loads(open(output_json_path.name).read())
 
     def _is_pass_code(self, code):

Modified: lnt/trunk/tests/runtest/test_suite.py
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/tests/runtest/test_suite.py?rev=262224&r1=262223&r2=262224&view=diff
==============================================================================
--- lnt/trunk/tests/runtest/test_suite.py (original)
+++ lnt/trunk/tests/runtest/test_suite.py Mon Feb 29 08:27:19 2016
@@ -286,3 +286,17 @@
 # CHECK-METRICS-DAG: foo.hash
 # CHECK-METRICS2-NOT: foo.unknown
 
+# Check that with a failing test, a report is still produced.
+# RUN: rm -f %t.SANDBOX/build/report.json
+# RUN: lnt runtest test-suite \
+# RUN:     --sandbox %t.SANDBOX \
+# RUN:     --no-timestamp \
+# RUN:     --no-configure \
+# RUN:     --test-suite %S/Inputs/test-suite-cmake \
+# RUN:     --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
+# RUN:     --use-cmake %S/Inputs/test-suite-cmake/fake-cmake \
+# RUN:     --use-make %S/Inputs/test-suite-cmake/fake-make \
+# RUN:     --use-lit %S/Inputs/test-suite-cmake/fake-lit-fails \
+# RUN:     --run-order=123 > %t.log 2> %t.err
+# RUN: FileCheck --check-prefix CHECK-RESULTS-FAIL < %t.SANDBOX/build/report.json %s
+# CHECK-RESULTS-FAIL: "run_order": "123"