[llvm] [llvm][llvm-lit] Add total time for each testsuite in JUnit XML output (PR #112230)
David Spickett via llvm-commits
llvm-commits at lists.llvm.org
Wed Oct 16 06:58:20 PDT 2024
DavidSpickett wrote:
> I will see what else is missing that is low friction to add. I suspect a lot of junit consumers are tolerant to missing fields.
I used a validator with the schema and found the following differences (aside from the testuite time I'm already adding here):
* We shouldn't be adding `time` to `<testsuites>`, but I don't see a reason to remove it. It is useful to have a total time to report.
* We do not add `id` to `<testsuite>`. This is an incrementing number from 0 that as far as I can tell, isn't that important. For example common tools to merge result files don't renumber this.
* We do not add `<system-out>` and `<system-err>` after each `<testcase>`. I think we bypass this by putting the output in the failure if the test does fail. We could improve this but it's not critical as long as failures are described.
* We do not add `<properties>` after `<testsuite>`. This is any environment variables and the like. We could try to add this but I think most people would look to the build system logs to find these anyway.
* We do not add `timestamp` to each `<testsuite>`, which is when the suite began. Could add, but not that useful compared to time taken.
* We do not add `hostname` to each `<testsuite>`. Lit is probably always on one host so this is not needed.
* We do not add `package` to each `<testsuite>`. This feels like a Java thing "Full class name of the test for non-aggregated testsuite documents.". I'm not sure what that would even be in our context.
* We do not add `errors` to `testsuite`. This is because JUnit draws a distinction between a failure and an error, an error might be more akin to lit's `UNRESOLVED`. However lit already considers `UNRESOLVED` to be a failure and that seems like a breaking change if we did that.
I could try to add `<testsuite timestamp=` but the value of that is low, especially for my use cases. The rest are not relevant to LLVM or risk being breaking changes.
https://github.com/llvm/llvm-project/pull/112230
More information about the llvm-commits
mailing list