[llvm] [ci] Handle the case where all reported tests pass but the build is still a failure (PR #120264)

[David Spickett via llvm-commits]

================
@@ -70,10 +72,51 @@ def test_no_failures(self):
             ),
         )
 
+    def test_no_failures_build_failed(self):
+        self.assertEqual(
+            _generate_report(
+                "Foo",
+                1,
+                [
+                    junit_from_xml(
+                        dedent(
+                            """\
+          <?xml version="1.0" encoding="UTF-8"?>
+          <testsuites time="0.00">
+          <testsuite name="Passed" tests="1" failures="0" skipped="0" time="0.00">
+          <testcase classname="Bar/test_1" name="test_1" time="0.00"/>
+          </testsuite>
+          </testsuites>"""
----------------
DavidSpickett wrote:

I'm not sure exactly what you're asking, perhaps I can explain the test a bit more and you tell me what parts of it you are referring to.

The test simulates a build where the final exit code was 1, but some tests ran and produced the JUnit XML shown here. That XML tells us that all tests that produced JUnit XML, passed. This JUnit XML comes straight from lit (in a production build), I'm not injecting a failure into it to account for the return code being non-zero.

(which you probably knew but stating it just in case)

So in that context, all this script has to handle is return code of 0 or 1, and JUnit XML has failures, or it doesn't. How the non-zero return code was created, it does not know.

If instead you are asking whether the script itself can handle JUnit XML failure records of different types, it's possible that there are some forms we would not detect, but when I wrote this script I referenced lit's tests for its own JUnit output. So chances are we can handle anything lit will produce.

You could also be wondering what happens if the build fails so badly that this script is run on files which are mangled, like if lit crashed half way through writing a file. This I had not thought of until now, and I will do another PR to handle that.

Or, does this test report try to capture results from tests that don't create JUnit XML, for example the libc's tests. It does not attempt to do that.

If a unittest binary failed with a signal, all this script would see is a non-zero return code. If there was no JUnit to process, this script wouldn't do anything and buildkite would mark the build as red automatically. If there was some JUnit, we'd make a report and if the JUnit listed no failures, we say so but still mark the report red as we know something else failed.

If it's not one of those scenarios, let me know which scenario you're thinking of.

https://github.com/llvm/llvm-project/pull/120264