[clang] [llvm] [ci] New script to generate test reports as Buildkite Annotations (PR #113447)

[David Spickett via cfe-commits]

DavidSpickett wrote:

I tried a few more things, so I will summarise everything up until this point. There are two layers here:
1. Making test reports.
2. Producing the XML test results.

Using the buildkite plugin is not an option (https://github.com/llvm/llvm-project/pull/113290) because we'd need docker or a fork of the plugin. Our own test reporting script is the only way to generate the reports. This decides `#1`, unless we want to leave it as is until the move to GitHub actions.

For `#2` the following ideas don't work:

* `tail ---follow --retry results_file > combined_results_file`, then splitting it up in the reporting script
  * Does not work because tail does not print the whole file on truncation. We would miss data.

* `mkfifo` and a listener script to write it out to unique files (https://github.com/llvm/llvm-project/pull/113703)
  * Works great on Linux but -
  * `mkfifo` fails silently in bash for Windows. There are no plans to fix that any time soon.
  * I could try https://learn.microsoft.com/en-us/windows/win32/ipc/named-pipes, but I suspect this would be even more code, and folks are already wary of adding too much to this system as it is.

* Using `ninja check-all`
  * Does not allow us to disable testing of a project, but still build it.
  * Does not help with runtimes tests that are separate builds.

These could work but have tradeoffs:

* Using the lit feature `--unique-output...` (this PR)
  * Built into lit (for now).
  * Minimal changes to scripts and pipeline.
    * New arguments for LLVM_LIT_ARGS.
    * Run the report generator on exit.
  * Risk of exposing an option publicly that we no longer need when the pipeline is re-written.
    * I could RFC that option and drum up interest in it from other users and external projects, if that would help. I have not done so up to now because it might have looked like I was trying to bias this discussion.

* Changing the build scripts to `ninja target` `mv result_file unique_name` in a loop (https://github.com/llvm/llvm-project/pull/113660)
  * The most script changes of any solution.
  * No changes to report generator.
  * Small bonus - result files are named after their check target (though I wonder if some check targets run lit multiple times, in which case this is actually a bug not a bonus).

* Wrapping lit in a Python script that will catch the output file name argument, run lit, then move the file to a unique name (https://github.com/llvm/llvm-project/pull/113896)
  * It works, albeit in a fiddly way because of differences in `.py` or not between Windows and Linux.
  * We have to re-wrap lit every time cmake is run (what I have implemented) or -
  * We configure in another directory, patch that lit, and configure with `-DLLVM_EXTERNAL_LIT=<path to the other lit>` (which I have yet to test)
  * We're not using the actual lit, though the chances of a regression because of that are pretty low. `ninja check-lit` does still work, so though we aren't testing what we ship, it's pretty close.

Are any of these solutions acceptable? Is there anything else I could do to address your concerns with any of these?

If folks think it's just too close to a move to GitHub Actions and full pipeline rewrite (which I fully support doing), then I will shelve all this. However, personally I'd like better test reporting even if it is only for 2/3 months until the new system appears.

https://github.com/llvm/llvm-project/pull/113447