[all-commits] [llvm/llvm-project] 1b199d: [ci] Handle the case where all reported tests pass...

[David Spickett via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 1b199d19902a752433c397377567ff381261e94a
      https://github.com/llvm/llvm-project/commit/1b199d19902a752433c397377567ff381261e94a
  Author: David Spickett <david.spickett at linaro.org>
  Date:   2025-01-13 (Mon, 13 Jan 2025)

  Changed paths:
    M .ci/generate_test_report.py
    M .ci/monolithic-linux.sh
    M .ci/monolithic-windows.sh

  Log Message:
  -----------
  [ci] Handle the case where all reported tests pass but the build is still a failure (#120264)

In this build:
https://buildkite.com/llvm-project/github-pull-requests/builds/126961

The builds actually failed, probably because prerequisite of a test
suite failed to build.

However they still ran other tests and all those passed. This meant that
the test reports were green even though the build was red. On some level
this is technically correct, but it is very misleading in practice.

So I've also passed the build script's return code, as it was when we
entered the on exit handler, to the generator, so that when this happens
again, the report will draw the viewer's attention to the overall
failure. There will be a link in the report to the build's log file, so
the next step to investigate is clear.

It would be nice to say "tests failed and there was some other build
error", but we cannot tell what the non-zero return code was caused by.
Could be either.

The script handles the following situations now:
| Have Result Files? | Tests reported failed? | Return code | Report |

|--------------------|------------------------|-------------|-----------------------------------------------------------------------------|
| Yes | No | 0 | Success style report. |
| Yes | Yes | 0 | Shouldn't happen, but if it did, failure style report
showing the failures. |
| Yes | No | 1 | Failure style report, showing no failures but noting
that the build failed. |
| Yes | Yes | 1 | Failure style report, showing the test failures. |
| No | ? | 0 | No test report, success shown in the normal build
display. |
| No | ? | 1 | No test report, failure shown in the normal build
display. |



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications