[llvm] [CI] Add link to issue tracker upon job failures (PR #140817)
Aiden Grossman via llvm-commits
llvm-commits at lists.llvm.org
Tue May 20 16:56:06 PDT 2025
https://github.com/boomanaiden154 updated https://github.com/llvm/llvm-project/pull/140817
>From 1821d24738e66ed7455e1da977692e94942ddf20 Mon Sep 17 00:00:00 2001
From: Aiden Grossman <aidengrossman at google.com>
Date: Tue, 20 May 2025 21:50:12 +0000
Subject: [PATCH 1/2] [CI] Add link to issue tracker upon job failures
The premerge system will fail somewhat often due to issues unrelated to
the patch being tested. This patch adds a link within the long form
outputs to the issue tracker prompting users to open an issue if they
see flakes/soemthing broken at HEAD/anything else wrong.
---
.ci/generate_test_report_lib.py | 10 +++++++++
.ci/generate_test_report_lib_test.py | 33 ++++++++++++++++++++--------
2 files changed, 34 insertions(+), 9 deletions(-)
diff --git a/.ci/generate_test_report_lib.py b/.ci/generate_test_report_lib.py
index 4ea25b26b761b..cbb44789b5370 100644
--- a/.ci/generate_test_report_lib.py
+++ b/.ci/generate_test_report_lib.py
@@ -122,6 +122,16 @@ def plural(num_tests):
]
)
+ if failures or return_code != 0:
+ report.extend(
+ [
+ "",
+ "If these failures are unrelated to your changes (e.g., tests are "
+ "broken or flaky at HEAD), please open an issue at "
+ "https://github.com/llvm/llvm-project/issues.",
+ ]
+ )
+
report = "\n".join(report)
if len(report.encode("utf-8")) > size_limit:
return generate_report(
diff --git a/.ci/generate_test_report_lib_test.py b/.ci/generate_test_report_lib_test.py
index fad6ab2bec899..14b1c3a6885c4 100644
--- a/.ci/generate_test_report_lib_test.py
+++ b/.ci/generate_test_report_lib_test.py
@@ -109,7 +109,9 @@ def test_no_failures_build_failed(self):
All tests passed but another part of the build **failed**.
- [Download](https://buildkite.com/organizations/organization_slug/pipelines/pipeline_slug/builds/build_number/jobs/job_id/download.txt) the build's log file to see the details."""
+ [Download](https://buildkite.com/organizations/organization_slug/pipelines/pipeline_slug/builds/build_number/jobs/job_id/download.txt) the build's log file to see the details.
+
+ If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues."""
),
"error",
),
@@ -169,7 +171,9 @@ def test_report_single_file_single_testsuite(self):
```
Other output goes here
```
- </details>"""
+ </details>
+
+ If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues."""
),
"error",
),
@@ -203,7 +207,9 @@ def test_report_single_file_single_testsuite(self):
```
DEF/test_2 output goes here
```
- </details>"""
+ </details>
+
+ If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues."""
),
"error",
)
@@ -311,7 +317,9 @@ def test_report_dont_list_failures(self):
* 1 test failed
- Failed tests and their output was too large to report. Download the build's log file to see the details."""
+ Failed tests and their output was too large to report. Download the build's log file to see the details.
+
+ If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues."""
),
"error",
),
@@ -352,13 +360,16 @@ def test_report_dont_list_failures_link_to_log(self):
* 1 test failed
- Failed tests and their output was too large to report. [Download](https://buildkite.com/organizations/organization_slug/pipelines/pipeline_slug/builds/build_number/jobs/job_id/download.txt) the build's log file to see the details."""
+ Failed tests and their output was too large to report. [Download](https://buildkite.com/organizations/organization_slug/pipelines/pipeline_slug/builds/build_number/jobs/job_id/download.txt) the build's log file to see the details.
+
+ If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues."""
),
"error",
),
)
def test_report_size_limit(self):
+ test_output = "f" * 1000
self.assertEqual(
generate_test_report_lib.generate_report(
"Foo",
@@ -371,14 +382,16 @@ def test_report_size_limit(self):
<testsuites time="0.02">
<testsuite name="Bar" tests="1" failures="1" skipped="0" time="0.02">
<testcase classname="Bar/test_1" name="test_1" time="0.02">
- <failure><![CDATA[Some long output goes here...]]></failure>
+ <failure><![CDATA[{output}]]></failure>
</testcase>
</testsuite>
- </testsuites>"""
+ </testsuites>""".format(
+ output=test_output
+ )
)
)
],
- size_limit=128,
+ size_limit=512,
),
(
dedent(
@@ -387,7 +400,9 @@ def test_report_size_limit(self):
* 1 test failed
- Failed tests and their output was too large to report. Download the build's log file to see the details."""
+ Failed tests and their output was too large to report. Download the build's log file to see the details.
+
+ If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues."""
),
"error",
),
>From 6f5e24e31695e6e830caea6ae3960ee7c7dfdf35 Mon Sep 17 00:00:00 2001
From: Aiden Grossman <aidengrossman at google.com>
Date: Tue, 20 May 2025 23:55:39 +0000
Subject: [PATCH 2/2] feedback
---
.ci/generate_test_report_lib.py | 3 ++-
.ci/generate_test_report_lib_test.py | 12 ++++++------
2 files changed, 8 insertions(+), 7 deletions(-)
diff --git a/.ci/generate_test_report_lib.py b/.ci/generate_test_report_lib.py
index cbb44789b5370..7023286d818bb 100644
--- a/.ci/generate_test_report_lib.py
+++ b/.ci/generate_test_report_lib.py
@@ -128,7 +128,8 @@ def plural(num_tests):
"",
"If these failures are unrelated to your changes (e.g., tests are "
"broken or flaky at HEAD), please open an issue at "
- "https://github.com/llvm/llvm-project/issues.",
+ "https://github.com/llvm/llvm-project/issues and add the "
+ "infrastructure label.",
]
)
diff --git a/.ci/generate_test_report_lib_test.py b/.ci/generate_test_report_lib_test.py
index 14b1c3a6885c4..f161653e5799c 100644
--- a/.ci/generate_test_report_lib_test.py
+++ b/.ci/generate_test_report_lib_test.py
@@ -111,7 +111,7 @@ def test_no_failures_build_failed(self):
[Download](https://buildkite.com/organizations/organization_slug/pipelines/pipeline_slug/builds/build_number/jobs/job_id/download.txt) the build's log file to see the details.
- If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues."""
+ If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the infrastructure label."""
),
"error",
),
@@ -173,7 +173,7 @@ def test_report_single_file_single_testsuite(self):
```
</details>
- If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues."""
+ If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the infrastructure label."""
),
"error",
),
@@ -209,7 +209,7 @@ def test_report_single_file_single_testsuite(self):
```
</details>
- If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues."""
+ If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the infrastructure label."""
),
"error",
)
@@ -319,7 +319,7 @@ def test_report_dont_list_failures(self):
Failed tests and their output was too large to report. Download the build's log file to see the details.
- If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues."""
+ If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the infrastructure label."""
),
"error",
),
@@ -362,7 +362,7 @@ def test_report_dont_list_failures_link_to_log(self):
Failed tests and their output was too large to report. [Download](https://buildkite.com/organizations/organization_slug/pipelines/pipeline_slug/builds/build_number/jobs/job_id/download.txt) the build's log file to see the details.
- If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues."""
+ If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the infrastructure label."""
),
"error",
),
@@ -402,7 +402,7 @@ def test_report_size_limit(self):
Failed tests and their output was too large to report. Download the build's log file to see the details.
- If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues."""
+ If these failures are unrelated to your changes (e.g., tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the infrastructure label."""
),
"error",
),
More information about the llvm-commits
mailing list