[llvm] [CI] Detect step failures in metrics job (PR #122564)

[Aiden Grossman via llvm-commits]

https://github.com/boomanaiden154 created https://github.com/llvm/llvm-project/pull/122564

This patch makes the metrics job also detect failures in individual steps. This is necessary now that we are setting continue-on-error in the premerge jobs to prevent sending out unnecessary email to detect what jobs actually fail.

>From 8a9c3eb9edcd803d354774e451a092c45c906439 Mon Sep 17 00:00:00 2001
From: Aiden Grossman <aidengrossman at google.com>
Date: Sat, 11 Jan 2025 02:19:02 +0000
Subject: [PATCH] [CI] Detect step failures in metrics job

This patch makes the metrics job also detect failures in individual steps.
This is necessary now that we are setting continue-on-error in the premerge
jobs to prevent sending out unnecessary email to detect what jobs actually
fail.
---
 .ci/metrics/metrics.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/.ci/metrics/metrics.py b/.ci/metrics/metrics.py
index 55025e50d1081f..bc46240dbd7fb5 100644
--- a/.ci/metrics/metrics.py
+++ b/.ci/metrics/metrics.py
@@ -80,6 +80,18 @@ def get_metrics(github_repo: github.Repository, workflows_to_track: dict[str, in
         completed_at = workflow_jobs[0].completed_at
 
         job_result = int(workflow_jobs[0].conclusion == "success")
+        if job_result:
+          # We still might want to mark the job as a failure if one of the steps
+          # failed. This is required due to use setting continue-on-error in
+          # the premerge pipeline to prevent sending emails while we are
+          # testing the infrastructure.
+          # TODO(boomanaiden154): Remove this once the premerge pipeline is no
+          # longer in a testing state and we can directly assert the workflow
+          # result.
+          for step in workflow_jobs[0].steps:
+            if step.conclusion != "success":
+              job_result = 0
+              break
 
         queue_time = started_at - created_at
         run_time = completed_at - started_at