[clang] [llvm] [ci] New script to generate test reports as Buildkite Annotations (PR #113447)

[David Spickett via cfe-commits]

DavidSpickett wrote:

I also looked through GitHub code search to see if anyone else was hitting this same issue.

ROCm avoids it by splitting each check target into its own step:
https://github.com/ROCm/ROCm/blob/bce439ecacc86b7f48934f5cc6be7ce2044412cf/.azuredevops/components/llvm-project.yml#L87

As we will in a future pipeline I'm sure.

The Direct X shader compiler uses one file but `check-all` instead:
https://github.com/microsoft/DirectXShaderCompiler/blob/6a7eae1bbe36d24471cb542fda12ec6a69f49572/azure-pipelines.yml#L116

Microsoft STL has a failure reporting script which has a subset of the output my proposed script produces:
https://github.com/microsoft/STL/blob/ca1553d38480b1224cb5b36ebd755af17aec275b/tools/scripts/print_failures.py

At one point the CHERI project called lit directly with all the test paths to avoid overwriting the results:
https://github.com/SchrodingerZhu/cheribuild/commit/36dbfbfb228afde8c2f0007575fe5f582fa4a652

In another place they used to use the `ninja / mv / ninja / mv ...` strategy:
https://github.com/CTSRD-CHERI/llvm-project/commit/cd8cdda1ede103d8cc9e0a2696e90cac429a7bc5#diff-8d76cd41051bde3e75bbe95280fde32df0252c12ee3e01b707a7c31c8f641320R32

The rest of the instances were Swift which seems to split checks into steps.

So there is evidence that this has been a problem, but that the solution has been to refactor the pipeline itself.
This gives you finer control over what builds, and you can have a report per step.

This supports reverting the lit option and choosing either `ninja / mv` or the lit wrapper script (assuming we don't shelve this until the GitHub Actions move).

https://github.com/llvm/llvm-project/pull/113447