[llvm] [llvm][llvm-lit] Add total time for each testsuite in JUnit XML output (PR #112230)

[David Spickett via llvm-commits]

https://github.com/DavidSpickett created https://github.com/llvm/llvm-project/pull/112230

Currently we write out a time taken to run all test suites:
```
<testsuites time="8.28">
```
And one for each test:
```
<testcase classname="lldb-shell.Breakpoint" name="breakpoint-command.test" time="2.38"/>
```
However, the schema says there should be one for each suite and test, but none for testsuites:
https://github.com/windyroad/JUnit-Schema/blob/cfa434d4b8e102a8f55b8727b552a0063ee9044e/JUnit.xsd#L216

I'm leaving the `testsuites` time in though because no one has complained so far, and someone out there probably has a script relying on it by now. Most XML tools handle unknown attributes quite well anyway.

I'm adding a per testsuite time to comply with the schema and maybe be more compatible with other JUnit tools.
```
<testsuite name="lldb-shell" ... time="12.34">
```

The test suite time is the sum of the time taken for all tests in the suite. This will ignore some overhead in setting up the suite, and means that the sum of the times for all individual suites may not equal the `testsuites` time.

As we're usually focusing on the execution time of particular tests, not lit's book keeping, I think this is a reasonable choice.

>From 79a1fabae2d3d66413fd86eb8f0ea64cb067d029 Mon Sep 17 00:00:00 2001
From: David Spickett <david.spickett at linaro.org>
Date: Mon, 14 Oct 2024 15:46:43 +0000
Subject: [PATCH] [llvm][llvm-lit] Add total time for each testsuite in JUnit
 XML output

Currently we write out a time taken to run all test suites:
<testsuites time="8.28">

And one for each test:
<testcase classname="lldb-shell.Breakpoint" name="breakpoint-command.test" time="2.38"/>

However, the schema says there should be one for each suite and test, but
none for testsuites:
https://github.com/windyroad/JUnit-Schema/blob/cfa434d4b8e102a8f55b8727b552a0063ee9044e/JUnit.xsd#L216

I'm leaving the testsuites time in though because no one has complained
so far, and someone out there probably has a script relying on it by now.
Most XML tools handle unknown attributes quite well anyway.

I'm adding a per testsuite time to comply with the schema and maybe
be more compatible with other JUnit tools.

<testsuite name="lldb-shell" ... time="12.34">

The test suite time is the sum of the time taken for all tests in the
suite. This will ignore some overhead in setting up the suite, and
means that the sum of the times for all invidual suites may not
equal the testsuites time.

As we're usually focusing on the execution time of particular tests,
not lit's book keeping, I think this is a reasonable choice.
---
 llvm/utils/lit/lit/reports.py         | 14 +++++++++++---
 llvm/utils/lit/tests/shtest-format.py |  2 +-
 llvm/utils/lit/tests/xunit-output.py  |  2 +-
 3 files changed, 13 insertions(+), 5 deletions(-)

diff --git a/llvm/utils/lit/lit/reports.py b/llvm/utils/lit/lit/reports.py
index 2ac44b0c0ce86e..d2d719b076bc70 100755
--- a/llvm/utils/lit/lit/reports.py
+++ b/llvm/utils/lit/lit/reports.py
@@ -105,12 +105,20 @@ def write_results(self, tests, elapsed):
             file.write("</testsuites>\n")
 
     def _write_testsuite(self, file, suite, tests):
-        skipped = sum(1 for t in tests if t.result.code in self.skipped_codes)
-        failures = sum(1 for t in tests if t.isFailure())
+        skipped = 0
+        failures = 0
+        time = 0.0
+
+        for t in tests:
+            if t.result.code in self.skipped_codes:
+                skipped += 1
+            if t.isFailure():
+                failures += 1
+            time += t.result.elapsed
 
         name = suite.config.name.replace(".", "-")
         file.write(
-            f'<testsuite name={quo(name)} tests="{len(tests)}" failures="{failures}" skipped="{skipped}">\n'
+            f'<testsuite name={quo(name)} tests="{len(tests)}" failures="{failures}" skipped="{skipped}" time="{time:.2f}">\n'
         )
         for test in tests:
             self._write_test(file, test, name)
diff --git a/llvm/utils/lit/tests/shtest-format.py b/llvm/utils/lit/tests/shtest-format.py
index 4a3d65b7bce4fd..3a1959549e5d01 100644
--- a/llvm/utils/lit/tests/shtest-format.py
+++ b/llvm/utils/lit/tests/shtest-format.py
@@ -107,7 +107,7 @@
 
 # XUNIT: <?xml version="1.0" encoding="UTF-8"?>
 # XUNIT-NEXT: <testsuites time="{{[0-9.]+}}">
-# XUNIT-NEXT: <testsuite name="shtest-format" tests="22" failures="8" skipped="3">
+# XUNIT-NEXT: <testsuite name="shtest-format" tests="22" failures="8" skipped="3" time="{{[0-9.]+}}">
 
 # XUNIT: <testcase classname="shtest-format.external_shell" name="fail.txt" time="{{[0-9]+\.[0-9]+}}">
 # XUNIT-NEXT: <failure{{[ ]*}}>
diff --git a/llvm/utils/lit/tests/xunit-output.py b/llvm/utils/lit/tests/xunit-output.py
index 67d99849fe36d9..392cded4653fe1 100644
--- a/llvm/utils/lit/tests/xunit-output.py
+++ b/llvm/utils/lit/tests/xunit-output.py
@@ -9,7 +9,7 @@
 
 # CHECK:      <?xml version="1.0" encoding="UTF-8"?>
 # CHECK-NEXT: <testsuites time="{{[0-9.]+}}">
-# CHECK-NEXT: <testsuite name="test-data" tests="5" failures="1" skipped="3">
+# CHECK-NEXT: <testsuite name="test-data" tests="5" failures="1" skipped="3" time="{{[0-9.]+}}">
 # CHECK-NEXT: <testcase classname="test-data.test-data" name="bad&name.ini" time="{{[0-1]\.[0-9]+}}">
 # CHECK-NEXT:   <failure><![CDATA[& < > ]]]]><![CDATA[> &"]]></failure>
 # CHECK-NEXT: </testcase>