[Release-testers] [llvm-dev] RFC: New Automated Release Workflow (using Issues and Pull Requests)
Mehdi AMINI via Release-testers
release-testers at lists.llvm.org
Mon Dec 20 18:21:49 PST 2021
On Mon, Dec 20, 2021 at 3:24 PM Tom Stellard <tstellar at redhat.com> wrote:
> On 12/20/21 09:16, Tom Stellard wrote:
> > On 12/18/21 15:04, David Blaikie wrote:
> >>
> >>
> >> On Fri, Dec 17, 2021 at 6:38 PM Tom Stellard <tstellar at redhat.com
> <mailto:tstellar at redhat.com>> wrote:
> >>
> >> On 12/17/21 16:47, David Blaikie wrote:
> >> > Sounds pretty good to me - wouldn't mind knowing more about/a
> good summary of the effects of this on project/repo/etc notifications that
> Mehdi's mentioning. (be good to have a write up of the expected
> impact/options to then discuss - from the thread so far I understand some
> general/high level concerns, but it's not clear to me exactly how it plays
> out)
> >> >
> >>
> >> The impact is really going to depend on the person and what
> notification preferences they
> >> have/want. If you are already watching the repo with the default
> settings, then you probably
> >> won't notice much of a difference given the current volume of
> notifications.
> >>
> >>
> >> I think I'm on the default settings - which does currently mean a
> notification for every issue update, which is a lot. Given that
> llvm-bugs at email.llvm.org <mailto:llvm-bugs at email.llvm.org> has been
> re-enabled, sending mail only on issue creation, I & others might opt back
> in to that behavior by disabling the baseline "notify on everything" to
> "notify only on issues I'm mentioned in".
> >>
> >> I guess currently the only email that github is generating is one email
> per issue update. We don't have any pull requests, so there aren't any
> emails for that, yeah?
> >>
> >> So this new strategy might add a few more back-and-forth on each
> cherrypick issue (for those using llvm-bugs & disabling general issue
> notifications, this will not be relevant to them - there won't be more
> issues created, just more comments on existing issues). But there will be
> some more emails generated related to the pull requests themselves, I
> guess? So each cherrypick goes from 2 emails to llvm-bugs (the issue
> creation and closure) to, how many? 4 (2 for llvm-bugs and I guess at least
> 2 for the pull request - one to make the request and one to close it -
> maybe a couple more status ones along the way?)
> >>
> >
> > I think the number of net new comments on issues will be very minimal or
> none at all. The
> > automated comments that are created by this process are replacing
> comments that I'm already making
> > manually.
> >
> > So 2+ for pull requests is probably a good estimate. I still need to
> figure out how many notifications
> > get generated for Actions with the default settings.
> >
>
> I did some research on the notifications and here is what I came up with:
>
> From what I can tell, notifications for actions are only sent to the
> user that initiated the event that led to the actions, so there would
> be no global notifications sent for the actions used by this workflow.
>
> There have been 131 bugs marked as release blockers in the llvm-13 cycle,
> this includes the 13.0.0 and 13.0.1 release. In the best case scenario,
> this proposal would generate 2 additional notifications per issue
> (1 for creating a pull request and 1 for merging it), and 0 net new
> issue comments (the automated comments just replace manual comments).
>
> If you assume that no manual comments would be replaced by the automation,
> then in the typical use case there would be a maximum of 4 notifications
> generated from issues (/cherry-pick comment, cherry-pick failed comment,
> /branch comment, /pull-request comment). In addition to the 2 pull
> request notifications.
>
> Based on this, my estimate is that this proposal will produce between
> (2 * 131) = 262 and (6 * 131) = 786 net new notifications every 6 months.
> Or between 1.46 and 4.367 net new notifications per day.
>
> For comparison, on Fri Dec 17, I received 115 email notifications from
> the llvm/llvm-project repo.
>
> The pull request emails should be easy for people to filter out of their
> inboxes with a rule. Pull request emails would have llvm/llvm-project in
> the To: field and have '(PR #123456)' at the end of the Subject: field
> (where 123456 is pull request number).
>
Actually it isn't enough: there isn't a way to filter on regexes in gmail
for example. Until GitHub allows the use of some different alias / target /
cc-email or similar mechanisms, it'll be hard to filter GitHub emails
accurately / reliably.
There are also the confusing aspects of starting to use pull-requests in
the monorepo, but only for some branches, which seem undesirable to me.
You didn't really elaborate about why not use a repo dedicated for managing
your actions and everything else? Seems like a conservative choice that
would provide isolation while still having the feature you desired,
wouldn't it?
>
> For people who filter out the pull request notifications, they would have
> between
> 0 and 2.9 net new notifications per day.
>
> - Tom
>
>
> > --Tom
> >
> >> If people want to give their notification preferences, I can try to
> look at how
> >> this change will impact specific configurations.
> >>
> >>
> >> @Mehdi AMINI <mailto:joker.eph at gmail.com> - are there particular
> scenarios you have in mind that'd be good to work through?
> >>
> >>
> >> -Tom
> >>
> >>
> >> > On Fri, Dec 17, 2021 at 1:15 PM Tom Stellard via llvm-dev <
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>> wrote:
> >> >
> >> > Hi,
> >> >
> >> > Here is a proposal for a new automated workflow for managing
> parts of the release
> >> > process. I've been experimenting with this over the past
> few releases and
> >> > now that we have migrated to GitHub issues, it would be
> possible for us to
> >> > implement this in the main repo.
> >> >
> >> > The workflow is pretty straight forward, but it does use
> pull requests. My
> >> > idea is to enable pull requests for only this automated
> workflow and not
> >> > for general development (i.e. We would still use Phabricator
> for code review).
> >> > Let me know what you think about this:
> >> >
> >> >
> >> > # Workflow
> >> >
> >> > * On an existing issue or a newly created issue, a user who
> wants to backport
> >> > one or more commits to the release branch adds a comment:
> >> >
> >> > /cherry-pick <commit_sha> <..>
> >> >
> >> > * This starts a GitHub Action job that attempts to
> cherry-pick the commit(s)
> >> > to the current release branch.
> >> >
> >> > * If the commit(s) can be cherry-picked cleanly, then the
> GitHub Action:
> >> > * Pushes the result of the cherry-pick to a branch in
> the
> >> > llvmbot/llvm-project repo called issue<n>, where n
> is the number of the
> >> > GitHub Issue that launched the Action.
> >> >
> >> > * Adds this comment on the issue: /branch
> llvmbot/llvm-project/issue<n>
> >> >
> >> > * Creates a pull request from
> llvmbot/llvm-project/issue<n> to
> >> > llvm/llvm-project/release/XX.x
> >> >
> >> > * Adds a comment on the issue: /pull-request #<n>
> >> > where n is the number of the pull request.
> >> >
> >> > * If the commit(s) can't be cherry-picked cleanly, then the
> GitHub Action job adds
> >> > the release:cherry-pick-failed label to the issue and adds a
> comment:
> >> > "Failed to cherry-pick <commit_sha> <..>" along with a link
> to the failing
> >> > Action.
> >> >
> >> > * If a user has manually cherry-picked the fixes, resolved
> the conflicts, and
> >> > pushed the result to a branch on github, they can
> automatically create a pull
> >> > request by adding this comment to an issue: /branch
> <user>/<repo>/<branch>
> >> >
> >> > * Once a pull request has been created, this launches more
> GitHub Actions
> >> > to run pre-commit tests.
> >> >
> >> > * Once the tests complete successfully and the changes have
> been approved
> >> > by the release manager, the pull request can me merged into
> the release branch.
> >> >
> >> > * After the pull request is merged, a GitHub Action
> automatically closes the
> >> > associated issue.
> >> >
> >> > Some Examples:
> >> >
> >> > Cherry-pick success:
> https://github.com/tstellar/llvm-project/issues/729 <
> https://github.com/tstellar/llvm-project/issues/729>
> >> > Cherry-pick <
> https://github.com/tstellar/llvm-project/issues/729Cherry-pick <
> https://github.com/tstellar/llvm-project/issues/729Cherry-pick>> failure:
> https://github.com/tstellar/llvm-project/issues/730 <
> https://github.com/tstellar/llvm-project/issues/730> <
> https://github.com/tstellar/llvm-project/issues/730 <
> https://github.com/tstellar/llvm-project/issues/730>>
> >> > Manual Branch comment:
> https://github.com/tstellar/llvm-project/issues/710 <
> https://github.com/tstellar/llvm-project/issues/710> <
> https://github.com/tstellar/llvm-project/issues/710 <
> https://github.com/tstellar/llvm-project/issues/710>>
> >> >
> >> >
> >> > # Motivation
> >> >
> >> > Why do this? The goal is to make the release process more
> efficient and transparent.
> >> > With this new workflow, users can get automatic and
> immediate feedback when a commit
> >> > they want backported doesn't apply cleanly or introduces
> some test failures. With
> >> > the current process, these kinds of issues are communicated
> by the release manager,
> >> > and it can be days or even weeks before a problem is
> discovered and communicated back
> >> > to the users.
> >> >
> >> > Another advantage of this workflow is it introduces
> pre-commit CI to the release branch,
> >> > which is important for the stability of the branch and the
> releases, but also gives
> >> > the project an opportunity to experiment with new CI
> workflows in a way that
> >> > does not disrupt development on the main branch.
> >> >
> >> > # Implementation
> >> >
> >> > If this proposal is accepted, I would plan to implement this
> for the LLVM 14 release cycle based
> >> > on the following proof of concept that I have been testing
> for the last few releases:
> >> >
> >> >
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml>
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
> >>
> >> >
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml>
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
> >>
> >> >
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml>
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
> >>
> >> >
> >> > Thanks,
> >> > Tom
> >> >
> >> > _______________________________________________
> >> > LLVM Developers mailing list
> >> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
> >> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> <
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>>
> >> >
> >>
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/release-testers/attachments/20211220/0a53c49b/attachment-0001.html>
More information about the Release-testers
mailing list