[clang] [llvm] [ci] New script to generate test reports as Buildkite Annotations (PR #113447)

[Aiden Grossman via cfe-commits]

boomanaiden154 wrote:

> I'd rather not do that. Commenting on the PR should be done very rarely, especially for CI failures, since it clutters the PR and Github already has a builtin system for presenting CI failures.

I think this is something that needs to be discussed further when we actually get there. Comments I think can still be useful in certain circumstances, but definitely agree that adding a comment for every push would clutter things, but that's definitely not the only way to do things.

> Overall, I'm fine with this patch, however I wonder how much effort it is worth putting into the BuildKite infrastructure given that we plan on moving as much as possible to Github Actions. 

The script is already written and works, and there still isn't a concrete timeline for when we are going to move things over to Github actions. This solves a valid complaint I hear somewhat often. For libc++ at least, there are definitely platforms that will not be able to move over to Github precommit where something like this might still be useful from what I understand.

> With Github actions, the .ci/generate-buildkite-pipeline-premerge script would probably go away entirely in favour of individual workflows or something along those lines, and then the issue of figuring out what failed would probably be a non-issue. I am especially unsure about adding functionality to fundamental utilities like Lit in that context: if we e.g. end up not needing to output test results in individual files a few weeks/months from now, then there may not be any users of --use-unique-output-file-name anymore and then we'll have increased the complexity of Lit for a temporary benefit.

Do we make any guarantees about supporting lit flags in the future? It's mostly an internal tool, and I would personally be in support of removing flags that are no longer used in tree, unless there are some extremely compelling downstream use cases and there is an appropriate maintenance story.

https://github.com/llvm/llvm-project/pull/113447