[llvm] [Flang] Github action to check for integration tests (PR #221766)
Kiran Chandramohan via llvm-commits
llvm-commits at lists.llvm.org
Tue Sep 22 14:08:09 PDT 2026
https://github.com/kiranchandramohan updated https://github.com/llvm/llvm-project/pull/221766
>From d3c8c390758f3c7dc97ea56cf9d8354ac8bd3a31 Mon Sep 17 00:00:00 2001
From: Kiran Chandramohan <kiran.chandramohan at arm.com>
Date: Fri, 21 Aug 2026 22:23:56 +0200
Subject: [PATCH 1/9] [Flang] Github action to check for integration tests
---
.../flang-integration-test-check.yml | 104 ++++++++++++++++++
1 file changed, 104 insertions(+)
create mode 100644 .github/workflows/flang-integration-test-check.yml
diff --git a/.github/workflows/flang-integration-test-check.yml b/.github/workflows/flang-integration-test-check.yml
new file mode 100644
index 0000000000000..452b650db0beb
--- /dev/null
+++ b/.github/workflows/flang-integration-test-check.yml
@@ -0,0 +1,104 @@
+name: Flang integration test check
+
+on:
+ pull_request_target:
+ types: [opened, synchronize, reopened]
+ paths:
+ - "flang/test/**"
+
+# This workflow only inspects file patches through the GitHub API. In
+# particular, it must not check out or execute code from the pull request.
+permissions:
+ contents: read
+ pull-requests: write
+
+jobs:
+ check-for-full-pipeline-tests:
+ runs-on: ubuntu-24.04
+ steps:
+ - name: Check for new full-pipeline test invocations
+ uses: actions/github-script at 3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+ with:
+ script: |
+ const marker = "<!-- flang-integration-test-check -->";
+ const pr = context.payload.pull_request;
+
+ const files = await github.paginate(
+ github.rest.pulls.listFiles,
+ {
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ pull_number: pr.number,
+ per_page: 100,
+ },
+ );
+
+ const findings = [];
+ for (const file of files) {
+ if (!file.filename.startsWith("flang/test/") || !file.patch)
+ continue;
+
+ // Only report newly added uses. Existing full-pipeline tests do
+ // not need to be justified again when an unrelated line changes.
+ const addsEmitLLVM = file.patch.split("\n").some(
+ line => /^\+(?!\+\+)/.test(line) && /\bRUN:/.test(line) &&
+ /(^|\s)-emit-llvm(?=\s|=|$)/.test(line),
+ );
+ if (addsEmitLLVM)
+ findings.push(`- \`${file.filename}\``);
+ }
+
+ const comments = await github.paginate(
+ github.rest.issues.listComments,
+ {
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ issue_number: pr.number,
+ per_page: 100,
+ },
+ );
+ const previous = comments.find(
+ comment => comment.user.type === "Bot" && comment.body.includes(marker),
+ );
+
+ if (findings.length === 0) {
+ if (previous) {
+ await github.rest.issues.deleteComment({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ comment_id: previous.id,
+ });
+ }
+ return;
+ }
+
+ const body = [
+ marker,
+ "This pull request adds a Flang test invocation that exercises the compiler " +
+ "pipeline through LLVM IR generation:",
+ "",
+ findings.join("\n"),
+ "",
+ "Please consider whether the same behavior can be covered by a focused test " +
+ "that stops at the relevant compiler stage. If LLVM IR generation is " +
+ "necessary, add a nearby, test-specific comment explaining why. See the " +
+ "[Flang integration test guidance](https://github.com/llvm/llvm-project/blob/main/flang/test/Integration/README.md).",
+ "",
+ "_This is an informational reminder; it does not fail the workflow._",
+ ].join("\n");
+
+ if (previous) {
+ await github.rest.issues.updateComment({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ comment_id: previous.id,
+ body,
+ });
+ } else {
+ await github.rest.issues.createComment({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ issue_number: pr.number,
+ body,
+ });
+ }
>From 11e14ee13fdebc29bd0a4f8f4c1d9320f60b91bb Mon Sep 17 00:00:00 2001
From: Kiran Chandramohan <kiran.chandramohan at arm.com>
Date: Tue, 22 Sep 2026 17:46:38 +0200
Subject: [PATCH 2/9] Address security warning and limit the action to the
llvm-project repo
---
.github/workflows/flang-integration-test-check.yml | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/.github/workflows/flang-integration-test-check.yml b/.github/workflows/flang-integration-test-check.yml
index 452b650db0beb..b03af46ee2caa 100644
--- a/.github/workflows/flang-integration-test-check.yml
+++ b/.github/workflows/flang-integration-test-check.yml
@@ -1,19 +1,23 @@
name: Flang integration test check
on:
- pull_request_target:
+ # This workflow needs pull_request_target so it can comment on pull requests
+ # from forks. It only examines patches using the GitHub API and never checks
+ # out or executes code from the pull request branch.
+ pull_request_target: # zizmor: ignore[dangerous-triggers]
types: [opened, synchronize, reopened]
paths:
- "flang/test/**"
-# This workflow only inspects file patches through the GitHub API. In
-# particular, it must not check out or execute code from the pull request.
permissions:
contents: read
- pull-requests: write
jobs:
check-for-full-pipeline-tests:
+ if: github.repository == 'llvm/llvm-project'
+ permissions:
+ contents: read
+ pull-requests: write
runs-on: ubuntu-24.04
steps:
- name: Check for new full-pipeline test invocations
>From 9d75cb2270bd72ad8e640f837e063812c19d179b Mon Sep 17 00:00:00 2001
From: Kiran Chandramohan <kiran.chandramohan at arm.com>
Date: Tue, 22 Sep 2026 19:25:46 +0200
Subject: [PATCH 3/9] [Flang] Reuse the shared PR comment workflow
Run the integration test check with read-only pull_request permissions
and upload comments for issue-write.yml. Update an existing reminder
when its findings disappear, and remove the pull_request_target suppression.
---
.../flang-integration-test-check.yml | 54 +++++++++----------
.github/workflows/issue-write.yml | 1 +
2 files changed, 25 insertions(+), 30 deletions(-)
diff --git a/.github/workflows/flang-integration-test-check.yml b/.github/workflows/flang-integration-test-check.yml
index b03af46ee2caa..bda5b2eaaa8e5 100644
--- a/.github/workflows/flang-integration-test-check.yml
+++ b/.github/workflows/flang-integration-test-check.yml
@@ -1,29 +1,26 @@
name: Flang integration test check
on:
- # This workflow needs pull_request_target so it can comment on pull requests
- # from forks. It only examines patches using the GitHub API and never checks
- # out or executes code from the pull request branch.
- pull_request_target: # zizmor: ignore[dangerous-triggers]
+ pull_request:
types: [opened, synchronize, reopened]
paths:
- "flang/test/**"
+ - ".github/workflows/flang-integration-test-check.yml"
permissions:
contents: read
+ pull-requests: read
jobs:
check-for-full-pipeline-tests:
if: github.repository == 'llvm/llvm-project'
- permissions:
- contents: read
- pull-requests: write
runs-on: ubuntu-24.04
steps:
- name: Check for new full-pipeline test invocations
uses: actions/github-script at 3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
+ const fs = require("fs");
const marker = "<!-- flang-integration-test-check -->";
const pr = context.payload.pull_request;
@@ -62,17 +59,19 @@ jobs:
},
);
const previous = comments.find(
- comment => comment.user.type === "Bot" && comment.body.includes(marker),
+ comment => comment.user.login === "github-actions[bot]" &&
+ comment.body.includes(marker),
);
+ // issue-write.yml creates or updates comments from this artifact.
+ // Resolve an existing reminder when its findings disappear.
if (findings.length === 0) {
- if (previous) {
- await github.rest.issues.deleteComment({
- owner: context.repo.owner,
- repo: context.repo.repo,
- comment_id: previous.id,
- });
- }
+ const resolved = previous ? [{
+ id: previous.id,
+ body: marker + "\n\nNo newly added `-emit-llvm` RUN lines remain " +
+ "under `flang/test/`.",
+ }] : [];
+ fs.writeFileSync("comments", JSON.stringify(resolved));
return;
}
@@ -91,18 +90,13 @@ jobs:
"_This is an informational reminder; it does not fail the workflow._",
].join("\n");
- if (previous) {
- await github.rest.issues.updateComment({
- owner: context.repo.owner,
- repo: context.repo.repo,
- comment_id: previous.id,
- body,
- });
- } else {
- await github.rest.issues.createComment({
- owner: context.repo.owner,
- repo: context.repo.repo,
- issue_number: pr.number,
- body,
- });
- }
+ const comment = { body };
+ if (previous)
+ comment.id = previous.id;
+ fs.writeFileSync("comments", JSON.stringify([comment]));
+
+ - name: Upload comment for issue-write.yml
+ uses: actions/upload-artifact at 043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+ with:
+ name: workflow-args
+ path: comments
diff --git a/.github/workflows/issue-write.yml b/.github/workflows/issue-write.yml
index 289ec207386e6..c7e1f2e4f13f1 100644
--- a/.github/workflows/issue-write.yml
+++ b/.github/workflows/issue-write.yml
@@ -11,6 +11,7 @@ on:
- "Test Issue Write"
- "Check LLVM ABI annotations"
- "Diff test-suite codegen"
+ - "Flang integration test check"
types:
- completed
>From a9d9a134f761dd0c83c80a915a5d0eb5a5b87241 Mon Sep 17 00:00:00 2001
From: Kiran Chandramohan <kiran.chandramohan at arm.com>
Date: Tue, 22 Sep 2026 20:09:15 +0200
Subject: [PATCH 4/9] [Flang] Run the integration test reminder for all Flang
changes
This is to avoid the situation where a subsequent change removes
the integration test from flang/test but the workflow would not
run because there are no changes in flang/test.
---
.github/workflows/flang-integration-test-check.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/flang-integration-test-check.yml b/.github/workflows/flang-integration-test-check.yml
index bda5b2eaaa8e5..6ca84fbf4d12a 100644
--- a/.github/workflows/flang-integration-test-check.yml
+++ b/.github/workflows/flang-integration-test-check.yml
@@ -4,7 +4,7 @@ on:
pull_request:
types: [opened, synchronize, reopened]
paths:
- - "flang/test/**"
+ - "flang/**"
- ".github/workflows/flang-integration-test-check.yml"
permissions:
>From 9fc932da920ee25ce6faf42830dc545981c2fe37 Mon Sep 17 00:00:00 2001
From: Kiran Chandramohan <kiran.chandramohan at arm.com>
Date: Tue, 22 Sep 2026 21:44:55 +0200
Subject: [PATCH 5/9] Remove registration of Flang integration test comment
workflow
This is done separately in #225500
---
.github/workflows/issue-write.yml | 1 -
1 file changed, 1 deletion(-)
diff --git a/.github/workflows/issue-write.yml b/.github/workflows/issue-write.yml
index c7e1f2e4f13f1..289ec207386e6 100644
--- a/.github/workflows/issue-write.yml
+++ b/.github/workflows/issue-write.yml
@@ -11,7 +11,6 @@ on:
- "Test Issue Write"
- "Check LLVM ABI annotations"
- "Diff test-suite codegen"
- - "Flang integration test check"
types:
- completed
>From f5ab1140e1442ce683a1e2c98a33fdaa5592c5e5 Mon Sep 17 00:00:00 2001
From: Kiran Chandramohan <kiran.chandramohan at arm.com>
Date: Tue, 22 Sep 2026 22:56:07 +0200
Subject: [PATCH 6/9] [Flang] Improve full-pipeline test detection and escape
filenames
Compare added and removed flag occurrences to avoid flagging edits to
existing invocations. Also recognize -emit-llvm-bc and -emit-obj, and
inspect DEFINE and REDEFINE directives alongside RUN lines.
Escape filenames in reminder comments to prevent Markdown injection.
---
.../flang-integration-test-check.yml | 31 ++++++++++++-------
1 file changed, 20 insertions(+), 11 deletions(-)
diff --git a/.github/workflows/flang-integration-test-check.yml b/.github/workflows/flang-integration-test-check.yml
index 6ca84fbf4d12a..10315d7a63029 100644
--- a/.github/workflows/flang-integration-test-check.yml
+++ b/.github/workflows/flang-integration-test-check.yml
@@ -39,14 +39,23 @@ jobs:
if (!file.filename.startsWith("flang/test/") || !file.patch)
continue;
- // Only report newly added uses. Existing full-pipeline tests do
- // not need to be justified again when an unrelated line changes.
- const addsEmitLLVM = file.patch.split("\n").some(
- line => /^\+(?!\+\+)/.test(line) && /\bRUN:/.test(line) &&
- /(^|\s)-emit-llvm(?=\s|=|$)/.test(line),
- );
- if (addsEmitLLVM)
- findings.push(`- \`${file.filename}\``);
+ // Count full-pipeline flag tokens across lit directives on both
+ // sides of the diff. Only report a net increase, so reindentation,
+ // option changes and moves into substitutions do not cause reminders.
+ // This is a heuristic, not a count of expanded invocations: a new
+ // use can be offset by a removed one elsewhere in the same file.
+ const directive = /\b(RUN|DEFINE|REDEFINE):/;
+ const fullPipeline = /(^|\s)(-emit-llvm(-bc)?|-emit-obj)(?=\s|=|$)/g;
+ const countFlags = line =>
+ directive.test(line) ? (line.match(fullPipeline) || []).length : 0;
+ const lines = file.patch.split("\n");
+ const total = sign => lines.filter(line => sign.test(line))
+ .reduce((n, line) => n + countFlags(line), 0);
+ if (total(/^\+(?!\+\+)/) > total(/^-(?!--)/)) {
+ // Keep filenames inside a single Markdown code span.
+ const filename = file.filename.replace(/[`\r\n]/g, "'");
+ findings.push(`- \`${filename}\``);
+ }
}
const comments = await github.paginate(
@@ -68,7 +77,7 @@ jobs:
if (findings.length === 0) {
const resolved = previous ? [{
id: previous.id,
- body: marker + "\n\nNo newly added `-emit-llvm` RUN lines remain " +
+ body: marker + "\n\nNo additional full-pipeline test invocations were detected " +
"under `flang/test/`.",
}] : [];
fs.writeFileSync("comments", JSON.stringify(resolved));
@@ -77,8 +86,8 @@ jobs:
const body = [
marker,
- "This pull request adds a Flang test invocation that exercises the compiler " +
- "pipeline through LLVM IR generation:",
+ "This pull request adds Flang test commands that exercise the compiler " +
+ "pipeline through LLVM IR generation or later stages:",
"",
findings.join("\n"),
"",
>From 04a1c1c5075455bcf7fce3861e963cc3233b6ee0 Mon Sep 17 00:00:00 2001
From: Kiran Chandramohan <kiran.chandramohan at arm.com>
Date: Tue, 22 Sep 2026 23:01:11 +0200
Subject: [PATCH 7/9] [Flang] Harden integration test reminder comment lookup
Guard against missing comment authors and non-string bodies. Require the marker at the start of a github-actions bot comment so quoted markers in unrelated comments are not selected for updates.
---
.github/workflows/flang-integration-test-check.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/flang-integration-test-check.yml b/.github/workflows/flang-integration-test-check.yml
index 10315d7a63029..f1ed033d8edb2 100644
--- a/.github/workflows/flang-integration-test-check.yml
+++ b/.github/workflows/flang-integration-test-check.yml
@@ -68,8 +68,8 @@ jobs:
},
);
const previous = comments.find(
- comment => comment.user.login === "github-actions[bot]" &&
- comment.body.includes(marker),
+ comment => comment.user && comment.user.login === "github-actions[bot]" &&
+ typeof comment.body === "string" && comment.body.startsWith(marker),
);
// issue-write.yml creates or updates comments from this artifact.
>From e6f1c72368098f312096c9dfed24b8e5d115e6f4 Mon Sep 17 00:00:00 2001
From: Kiran Chandramohan <kiran.chandramohan at arm.com>
Date: Tue, 22 Sep 2026 23:03:36 +0200
Subject: [PATCH 8/9] [Flang] Log test files with missing API patches
Emit a workflow notice when a Flang test file has no patch text to inspect, making skipped files visible in the run log.
---
.github/workflows/flang-integration-test-check.yml | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/flang-integration-test-check.yml b/.github/workflows/flang-integration-test-check.yml
index f1ed033d8edb2..b48e17a850546 100644
--- a/.github/workflows/flang-integration-test-check.yml
+++ b/.github/workflows/flang-integration-test-check.yml
@@ -36,8 +36,12 @@ jobs:
const findings = [];
for (const file of files) {
- if (!file.filename.startsWith("flang/test/") || !file.patch)
+ if (!file.filename.startsWith("flang/test/"))
continue;
+ if (!file.patch) {
+ core.notice(`Not checked (no patch text from the API): ${file.filename}`);
+ continue;
+ }
// Count full-pipeline flag tokens across lit directives on both
// sides of the diff. Only report a net increase, so reindentation,
>From 34c0d9f3c7040b599067f61068b306b6c0880b8f Mon Sep 17 00:00:00 2001
From: Kiran Chandramohan <kiran.chandramohan at arm.com>
Date: Tue, 22 Sep 2026 23:03:41 +0200
Subject: [PATCH 9/9] [Flang] Limit the integration test check to ten minutes
---
.github/workflows/flang-integration-test-check.yml | 1 +
1 file changed, 1 insertion(+)
diff --git a/.github/workflows/flang-integration-test-check.yml b/.github/workflows/flang-integration-test-check.yml
index b48e17a850546..de1903d60247b 100644
--- a/.github/workflows/flang-integration-test-check.yml
+++ b/.github/workflows/flang-integration-test-check.yml
@@ -15,6 +15,7 @@ jobs:
check-for-full-pipeline-tests:
if: github.repository == 'llvm/llvm-project'
runs-on: ubuntu-24.04
+ timeout-minutes: 10
steps:
- name: Check for new full-pipeline test invocations
uses: actions/github-script at 3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
More information about the llvm-commits
mailing list