[Release-testers] [llvm-dev] RFC: New Automated Release Workflow (using Issues and Pull Requests)
Mehdi AMINI via Release-testers
release-testers at lists.llvm.org
Wed Dec 22 19:51:04 PST 2021
On Mon, Dec 20, 2021 at 9:18 PM Tom Stellard <tstellar at redhat.com> wrote:
> On 12/20/21 18:21, Mehdi AMINI wrote:
> >
> >
> > On Mon, Dec 20, 2021 at 3:24 PM Tom Stellard <tstellar at redhat.com
> <mailto:tstellar at redhat.com>> wrote:
> >
> > On 12/20/21 09:16, Tom Stellard wrote:
> > > On 12/18/21 15:04, David Blaikie wrote:
> > >>
> > >>
> > >> On Fri, Dec 17, 2021 at 6:38 PM Tom Stellard <
> tstellar at redhat.com <mailto:tstellar at redhat.com> <mailto:
> tstellar at redhat.com <mailto:tstellar at redhat.com>>> wrote:
> > >>
> > >> On 12/17/21 16:47, David Blaikie wrote:
> > >> > Sounds pretty good to me - wouldn't mind knowing more
> about/a good summary of the effects of this on project/repo/etc
> notifications that Mehdi's mentioning. (be good to have a write up of the
> expected impact/options to then discuss - from the thread so far I
> understand some general/high level concerns, but it's not clear to me
> exactly how it plays out)
> > >> >
> > >>
> > >> The impact is really going to depend on the person and what
> notification preferences they
> > >> have/want. If you are already watching the repo with the
> default settings, then you probably
> > >> won't notice much of a difference given the current volume
> of notifications.
> > >>
> > >>
> > >> I think I'm on the default settings - which does currently mean
> a notification for every issue update, which is a lot. Given that
> llvm-bugs at email.llvm.org <mailto:llvm-bugs at email.llvm.org> <mailto:
> llvm-bugs at email.llvm.org <mailto:llvm-bugs at email.llvm.org>> has been
> re-enabled, sending mail only on issue creation, I & others might opt back
> in to that behavior by disabling the baseline "notify on everything" to
> "notify only on issues I'm mentioned in".
> > >>
> > >> I guess currently the only email that github is generating is
> one email per issue update. We don't have any pull requests, so there
> aren't any emails for that, yeah?
> > >>
> > >> So this new strategy might add a few more back-and-forth on each
> cherrypick issue (for those using llvm-bugs & disabling general issue
> notifications, this will not be relevant to them - there won't be more
> issues created, just more comments on existing issues). But there will be
> some more emails generated related to the pull requests themselves, I
> guess? So each cherrypick goes from 2 emails to llvm-bugs (the issue
> creation and closure) to, how many? 4 (2 for llvm-bugs and I guess at least
> 2 for the pull request - one to make the request and one to close it -
> maybe a couple more status ones along the way?)
> > >>
> > >
> > > I think the number of net new comments on issues will be very
> minimal or none at all. The
> > > automated comments that are created by this process are replacing
> comments that I'm already making
> > > manually.
> > >
> > > So 2+ for pull requests is probably a good estimate. I still
> need to figure out how many notifications
> > > get generated for Actions with the default settings.
> > >
> >
> > I did some research on the notifications and here is what I came up
> with:
> >
> > From what I can tell, notifications for actions are only sent to
> the
> > user that initiated the event that led to the actions, so there would
> > be no global notifications sent for the actions used by this
> workflow.
> >
> > There have been 131 bugs marked as release blockers in the llvm-13
> cycle,
> > this includes the 13.0.0 and 13.0.1 release. In the best case
> scenario,
> > this proposal would generate 2 additional notifications per issue
> > (1 for creating a pull request and 1 for merging it), and 0 net new
> > issue comments (the automated comments just replace manual comments).
> >
> > If you assume that no manual comments would be replaced by the
> automation,
> > then in the typical use case there would be a maximum of 4
> notifications
> > generated from issues (/cherry-pick comment, cherry-pick failed
> comment,
> > /branch comment, /pull-request comment). In addition to the 2 pull
> > request notifications.
> >
> > Based on this, my estimate is that this proposal will produce between
> > (2 * 131) = 262 and (6 * 131) = 786 net new notifications every 6
> months.
> > Or between 1.46 and 4.367 net new notifications per day.
> >
> > For comparison, on Fri Dec 17, I received 115 email notifications
> from
> > the llvm/llvm-project repo.
> >
> > The pull request emails should be easy for people to filter out of
> their
> > inboxes with a rule. Pull request emails would have
> llvm/llvm-project in
> > the To: field and have '(PR #123456)' at the end of the Subject:
> field
> > (where 123456 is pull request number).
> >
> >
> > Actually it isn't enough: there isn't a way to filter on regexes in
> gmail for example. Until GitHub allows the use of some different alias /
> target / cc-email or similar mechanisms, it'll be hard to filter GitHub
> emails accurately / reliably.
> >
>
> Matching on '(PR #' might be enough if people wanted to try it.
>
Gmail ignores punctuation unfortunately, this is partly why the current
situation with issue notification is also quite messy and hard to manage
on the client side.
>
> > There are also the confusing aspects of starting to use pull-requests in
> the monorepo, but only for some branches, which seem undesirable to me.
> >
>
> Yeah, this is one of the downsides of using pull-requests in the monorepo.
>
> > You didn't really elaborate about why not use a repo dedicated for
> managing your actions and everything else? Seems like a conservative choice
> that would provide isolation while still having the feature you desired,
> wouldn't it?
> >
>
> One advantage of using the monorepo for the pull-requests is that it
> simplifies
> the workflow. With pull-requests in a secondary repo, there would have to
> be an
> additional step to sync from the secondary repo into the monorepo after the
> pull-request was merged. You would also need to setup a sync from the
> monorepo
> to the secondary repo in case something was pushed directly to the
> monorepo.
>
> Using the monorepo for pull requests also makes it easier to enable
> self-hosted
> action runners, because you would only need to set them up for one repo and
> wouldn't have to figure out how to coordinate their usage between two.
> And if we were able to get GitHub to give us access to some of their more
> powerful
> runners, it might be more difficult to enable them for both repos.
>
I would hope that we can get access to this for the entire organization and
just for one repo.
--
Mehdi
>
> - Tom
>
> >
> >
> > For people who filter out the pull request notifications, they would
> have between
> > 0 and 2.9 net new notifications per day.
> >
> > - Tom
> >
> >
> > > --Tom
> > >
> > >> If people want to give their notification preferences, I can
> try to look at how
> > >> this change will impact specific configurations.
> > >>
> > >>
> > >> @Mehdi AMINI <mailto:joker.eph at gmail.com <mailto:
> joker.eph at gmail.com>> - are there particular scenarios you have in mind
> that'd be good to work through?
> > >>
> > >>
> > >> -Tom
> > >>
> > >>
> > >> > On Fri, Dec 17, 2021 at 1:15 PM Tom Stellard via llvm-dev
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> <mailto:
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>>> wrote:
> > >> >
> > >> > Hi,
> > >> >
> > >> > Here is a proposal for a new automated workflow for
> managing parts of the release
> > >> > process. I've been experimenting with this over the
> past few releases and
> > >> > now that we have migrated to GitHub issues, it would
> be possible for us to
> > >> > implement this in the main repo.
> > >> >
> > >> > The workflow is pretty straight forward, but it does
> use pull requests. My
> > >> > idea is to enable pull requests for only this
> automated workflow and not
> > >> > for general development (i.e. We would still use
> Phabricator for code review).
> > >> > Let me know what you think about this:
> > >> >
> > >> >
> > >> > # Workflow
> > >> >
> > >> > * On an existing issue or a newly created issue, a
> user who wants to backport
> > >> > one or more commits to the release branch adds a
> comment:
> > >> >
> > >> > /cherry-pick <commit_sha> <..>
> > >> >
> > >> > * This starts a GitHub Action job that attempts to
> cherry-pick the commit(s)
> > >> > to the current release branch.
> > >> >
> > >> > * If the commit(s) can be cherry-picked cleanly, then
> the GitHub Action:
> > >> > * Pushes the result of the cherry-pick to a
> branch in the
> > >> > llvmbot/llvm-project repo called issue<n>,
> where n is the number of the
> > >> > GitHub Issue that launched the Action.
> > >> >
> > >> > * Adds this comment on the issue: /branch
> llvmbot/llvm-project/issue<n>
> > >> >
> > >> > * Creates a pull request from
> llvmbot/llvm-project/issue<n> to
> > >> > llvm/llvm-project/release/XX.x
> > >> >
> > >> > * Adds a comment on the issue: /pull-request
> #<n>
> > >> > where n is the number of the pull request.
> > >> >
> > >> > * If the commit(s) can't be cherry-picked cleanly,
> then the GitHub Action job adds
> > >> > the release:cherry-pick-failed label to the issue and
> adds a comment:
> > >> > "Failed to cherry-pick <commit_sha> <..>" along with
> a link to the failing
> > >> > Action.
> > >> >
> > >> > * If a user has manually cherry-picked the fixes,
> resolved the conflicts, and
> > >> > pushed the result to a branch on github, they can
> automatically create a pull
> > >> > request by adding this comment to an issue: /branch
> <user>/<repo>/<branch>
> > >> >
> > >> > * Once a pull request has been created, this launches
> more GitHub Actions
> > >> > to run pre-commit tests.
> > >> >
> > >> > * Once the tests complete successfully and the
> changes have been approved
> > >> > by the release manager, the pull request can me
> merged into the release branch.
> > >> >
> > >> > * After the pull request is merged, a GitHub Action
> automatically closes the
> > >> > associated issue.
> > >> >
> > >> > Some Examples:
> > >> >
> > >> > Cherry-pick success:
> https://github.com/tstellar/llvm-project/issues/729 <
> https://github.com/tstellar/llvm-project/issues/729> <
> https://github.com/tstellar/llvm-project/issues/729 <
> https://github.com/tstellar/llvm-project/issues/729>>
> > >> > Cherry-pick <
> https://github.com/tstellar/llvm-project/issues/729Cherry-pick <
> https://github.com/tstellar/llvm-project/issues/729Cherry-pick> <
> https://github.com/tstellar/llvm-project/issues/729Cherry-pick <
> https://github.com/tstellar/llvm-project/issues/729Cherry-pick>>>
> failure: https://github.com/tstellar/llvm-project/issues/730 <
> https://github.com/tstellar/llvm-project/issues/730> <
> https://github.com/tstellar/llvm-project/issues/730 <
> https://github.com/tstellar/llvm-project/issues/730>> <
> https://github.com/tstellar/llvm-project/issues/730 <
> https://github.com/tstellar/llvm-project/issues/730> <
> https://github.com/tstellar/llvm-project/issues/730 <
> https://github.com/tstellar/llvm-project/issues/730>>>
> > >> > Manual Branch comment:
> https://github.com/tstellar/llvm-project/issues/710 <
> https://github.com/tstellar/llvm-project/issues/710> <
> https://github.com/tstellar/llvm-project/issues/710 <
> https://github.com/tstellar/llvm-project/issues/710>> <
> https://github.com/tstellar/llvm-project/issues/710 <
> https://github.com/tstellar/llvm-project/issues/710> <
> https://github.com/tstellar/llvm-project/issues/710 <
> https://github.com/tstellar/llvm-project/issues/710>>>
> > >> >
> > >> >
> > >> > # Motivation
> > >> >
> > >> > Why do this? The goal is to make the release process
> more efficient and transparent.
> > >> > With this new workflow, users can get automatic and
> immediate feedback when a commit
> > >> > they want backported doesn't apply cleanly or
> introduces some test failures. With
> > >> > the current process, these kinds of issues are
> communicated by the release manager,
> > >> > and it can be days or even weeks before a problem is
> discovered and communicated back
> > >> > to the users.
> > >> >
> > >> > Another advantage of this workflow is it introduces
> pre-commit CI to the release branch,
> > >> > which is important for the stability of the branch
> and the releases, but also gives
> > >> > the project an opportunity to experiment with new CI
> workflows in a way that
> > >> > does not disrupt development on the main branch.
> > >> >
> > >> > # Implementation
> > >> >
> > >> > If this proposal is accepted, I would plan to
> implement this for the LLVM 14 release cycle based
> > >> > on the following proof of concept that I have been
> testing for the last few releases:
> > >> >
> > >> >
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml>
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml>>
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml>
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
> >>>
> > >> >
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml>
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml>>
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml>
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
> >>>
> > >> >
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml>
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml>>
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml>
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
> <
> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
> >>>
> > >> >
> > >> > Thanks,
> > >> > Tom
> > >> >
> > >> > _______________________________________________
> > >> > LLVM Developers mailing list
> > >> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> <mailto:
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>>
> > >> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> <
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>> <
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> <
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>>>
> > >> >
> > >>
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/release-testers/attachments/20211222/01f0c4f9/attachment-0001.html>
More information about the Release-testers
mailing list