[llvm] 0231227 - Reapply "[lit] GoogleTest framework should report failures if test binary crashes"

[Stephen Neuendorffer via llvm-commits]

Author: Stephen Neuendorffer
Date: 2020-05-22T14:01:08-07:00
New Revision: 0231227e5d8a7c4737ce2316dae01b63a2adfbe5

URL: https://github.com/llvm/llvm-project/commit/0231227e5d8a7c4737ce2316dae01b63a2adfbe5
DIFF: https://github.com/llvm/llvm-project/commit/0231227e5d8a7c4737ce2316dae01b63a2adfbe5.diff

LOG: Reapply "[lit] GoogleTest framework should report failures if test binary crashes"

This reverts commit 78dea0e8fb6cc5c0fae64d65b7f40560cab7b329.

The offending lldb test (which is a real bug exposed by this patch)
has been disabled on windows (see a67b2faa7c4cfbceffb4213f46769c45a5a9291a)
and lldb is queued for inclusion into precommit testing, which would
have caught this.

Differential Revision: https://reviews.llvm.org/D80389

Added: 
    llvm/utils/lit/tests/Inputs/googletest-discovery-failed/lit.cfg
    llvm/utils/lit/tests/Inputs/googletest-discovery-failed/subdir/OneTest.py
    llvm/utils/lit/tests/googletest-discovery-failed.py

Modified: 
    llvm/utils/lit/lit/formats/googletest.py

Removed: 
    


################################################################################
diff  --git a/llvm/utils/lit/lit/formats/googletest.py b/llvm/utils/lit/lit/formats/googletest.py
index 67f32cf1cdf7..059eb99762e9 100644
--- a/llvm/utils/lit/lit/formats/googletest.py
+++ b/llvm/utils/lit/lit/formats/googletest.py
@@ -41,6 +41,14 @@ def getGTestTests(self, path, litConfig, localConfig):
             litConfig.warning(
                 "unable to discover google-tests in %r: %s. Process output: %s"
                 % (path, sys.exc_info()[1], exc.output))
+            # This doesn't look like a valid gtest file.  This can
+            # have a number of causes, none of them good.  For
+            # instance, we could have created a broken executable.
+            # Alternatively, someone has cruft in their test
+            # directory.  If we don't return a test here, then no
+            # failures will get reported, so return a dummy test name
+            # so that the failure is reported later.
+            yield 'failed_to_discover_tests_from_gtest'
             return
 
         nested_tests = []

diff  --git a/llvm/utils/lit/tests/Inputs/googletest-discovery-failed/lit.cfg b/llvm/utils/lit/tests/Inputs/googletest-discovery-failed/lit.cfg
new file mode 100644
index 000000000000..66be8ec33388
--- /dev/null
+++ b/llvm/utils/lit/tests/Inputs/googletest-discovery-failed/lit.cfg
@@ -0,0 +1,3 @@
+import lit.formats
+config.name = 'googletest-discovery-failed'
+config.test_format = lit.formats.GoogleTest('subdir', 'Test')

diff  --git a/llvm/utils/lit/tests/Inputs/googletest-discovery-failed/subdir/OneTest.py b/llvm/utils/lit/tests/Inputs/googletest-discovery-failed/subdir/OneTest.py
new file mode 100644
index 000000000000..4816459dee45
--- /dev/null
+++ b/llvm/utils/lit/tests/Inputs/googletest-discovery-failed/subdir/OneTest.py
@@ -0,0 +1,3 @@
+#!/usr/bin/env python
+
+raise SystemExit("We are not a valid gtest executable!")

diff  --git a/llvm/utils/lit/tests/googletest-discovery-failed.py b/llvm/utils/lit/tests/googletest-discovery-failed.py
new file mode 100644
index 000000000000..a8b7921f1571
--- /dev/null
+++ b/llvm/utils/lit/tests/googletest-discovery-failed.py
@@ -0,0 +1,10 @@
+# Check for correct error message when discovery of tests fails.
+#
+# RUN: not %{lit} -j 1 -v %{inputs}/googletest-discovery-failed > %t.cmd.out
+# RUN: FileCheck < %t.cmd.out %s
+
+
+# CHECK: -- Testing:
+# CHECK: Failing Tests (1):
+# CHECK:   googletest-discovery-failed :: subdir/OneTest.py/failed_to_discover_tests_from_gtest
+# CHECK: Unexpected Failures: 1