[llvm-dev] RFC: New Automated Release Workflow (using Issues and Pull Requests)
Philip Reames via llvm-dev
llvm-dev at lists.llvm.org
Mon Dec 20 15:34:15 PST 2021
On 12/20/21 3:24 PM, Tom Stellard via llvm-dev wrote:
> On 12/20/21 09:16, Tom Stellard wrote:
>> On 12/18/21 15:04, David Blaikie wrote:
>>>
>>>
>>> On Fri, Dec 17, 2021 at 6:38 PM Tom Stellard <tstellar at redhat.com
>>> <mailto:tstellar at redhat.com>> wrote:
>>>
>>> On 12/17/21 16:47, David Blaikie wrote:
>>> > Sounds pretty good to me - wouldn't mind knowing more about/a
>>> good summary of the effects of this on project/repo/etc
>>> notifications that Mehdi's mentioning. (be good to have a write up
>>> of the expected impact/options to then discuss - from the thread so
>>> far I understand some general/high level concerns, but it's not
>>> clear to me exactly how it plays out)
>>> >
>>>
>>> The impact is really going to depend on the person and what
>>> notification preferences they
>>> have/want. If you are already watching the repo with the
>>> default settings, then you probably
>>> won't notice much of a difference given the current volume of
>>> notifications.
>>>
>>>
>>> I think I'm on the default settings - which does currently mean a
>>> notification for every issue update, which is a lot. Given that
>>> llvm-bugs at email.llvm.org <mailto:llvm-bugs at email.llvm.org> has been
>>> re-enabled, sending mail only on issue creation, I & others might
>>> opt back in to that behavior by disabling the baseline "notify on
>>> everything" to "notify only on issues I'm mentioned in".
>>>
>>> I guess currently the only email that github is generating is one
>>> email per issue update. We don't have any pull requests, so there
>>> aren't any emails for that, yeah?
>>>
>>> So this new strategy might add a few more back-and-forth on each
>>> cherrypick issue (for those using llvm-bugs & disabling general
>>> issue notifications, this will not be relevant to them - there won't
>>> be more issues created, just more comments on existing issues). But
>>> there will be some more emails generated related to the pull
>>> requests themselves, I guess? So each cherrypick goes from 2 emails
>>> to llvm-bugs (the issue creation and closure) to, how many? 4 (2 for
>>> llvm-bugs and I guess at least 2 for the pull request - one to make
>>> the request and one to close it - maybe a couple more status ones
>>> along the way?)
>>>
>>
>> I think the number of net new comments on issues will be very minimal
>> or none at all. The
>> automated comments that are created by this process are replacing
>> comments that I'm already making
>> manually.
>>
>> So 2+ for pull requests is probably a good estimate. I still need to
>> figure out how many notifications
>> get generated for Actions with the default settings.
>>
>
> I did some research on the notifications and here is what I came up with:
>
> From what I can tell, notifications for actions are only sent to the
> user that initiated the event that led to the actions, so there would
> be no global notifications sent for the actions used by this workflow.
>
> There have been 131 bugs marked as release blockers in the llvm-13 cycle,
> this includes the 13.0.0 and 13.0.1 release. In the best case scenario,
> this proposal would generate 2 additional notifications per issue
> (1 for creating a pull request and 1 for merging it), and 0 net new
> issue comments (the automated comments just replace manual comments).
>
> If you assume that no manual comments would be replaced by the
> automation,
> then in the typical use case there would be a maximum of 4 notifications
> generated from issues (/cherry-pick comment, cherry-pick failed comment,
> /branch comment, /pull-request comment). In addition to the 2 pull
> request notifications.
>
> Based on this, my estimate is that this proposal will produce between
> (2 * 131) = 262 and (6 * 131) = 786 net new notifications every 6 months.
> Or between 1.46 and 4.367 net new notifications per day.
>
> For comparison, on Fri Dec 17, I received 115 email notifications from
> the llvm/llvm-project repo.
>
> The pull request emails should be easy for people to filter out of their
> inboxes with a rule. Pull request emails would have llvm/llvm-project in
> the To: field and have '(PR #123456)' at the end of the Subject: field
> (where 123456 is pull request number).
>
> For people who filter out the pull request notifications, they would
> have between
> 0 and 2.9 net new notifications per day.
This seems both fairly minimal, and well justified to me.
Philip
>
> - Tom
>
>
>> --Tom
>>
>>> If people want to give their notification preferences, I can try
>>> to look at how
>>> this change will impact specific configurations.
>>>
>>>
>>> @Mehdi AMINI <mailto:joker.eph at gmail.com> - are there particular
>>> scenarios you have in mind that'd be good to work through?
>>>
>>>
>>> -Tom
>>>
>>>
>>> > On Fri, Dec 17, 2021 at 1:15 PM Tom Stellard via llvm-dev
>>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>>> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>>
>>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > Here is a proposal for a new automated workflow for
>>> managing parts of the release
>>> > process. I've been experimenting with this over the past
>>> few releases and
>>> > now that we have migrated to GitHub issues, it would be
>>> possible for us to
>>> > implement this in the main repo.
>>> >
>>> > The workflow is pretty straight forward, but it does use
>>> pull requests. My
>>> > idea is to enable pull requests for only this automated
>>> workflow and not
>>> > for general development (i.e. We would still use
>>> Phabricator for code review).
>>> > Let me know what you think about this:
>>> >
>>> >
>>> > # Workflow
>>> >
>>> > * On an existing issue or a newly created issue, a user
>>> who wants to backport
>>> > one or more commits to the release branch adds a comment:
>>> >
>>> > /cherry-pick <commit_sha> <..>
>>> >
>>> > * This starts a GitHub Action job that attempts to
>>> cherry-pick the commit(s)
>>> > to the current release branch.
>>> >
>>> > * If the commit(s) can be cherry-picked cleanly, then the
>>> GitHub Action:
>>> > * Pushes the result of the cherry-pick to a branch
>>> in the
>>> > llvmbot/llvm-project repo called issue<n>, where
>>> n is the number of the
>>> > GitHub Issue that launched the Action.
>>> >
>>> > * Adds this comment on the issue: /branch
>>> llvmbot/llvm-project/issue<n>
>>> >
>>> > * Creates a pull request from
>>> llvmbot/llvm-project/issue<n> to
>>> > llvm/llvm-project/release/XX.x
>>> >
>>> > * Adds a comment on the issue: /pull-request #<n>
>>> > where n is the number of the pull request.
>>> >
>>> > * If the commit(s) can't be cherry-picked cleanly, then
>>> the GitHub Action job adds
>>> > the release:cherry-pick-failed label to the issue and
>>> adds a comment:
>>> > "Failed to cherry-pick <commit_sha> <..>" along with a
>>> link to the failing
>>> > Action.
>>> >
>>> > * If a user has manually cherry-picked the fixes,
>>> resolved the conflicts, and
>>> > pushed the result to a branch on github, they can
>>> automatically create a pull
>>> > request by adding this comment to an issue: /branch
>>> <user>/<repo>/<branch>
>>> >
>>> > * Once a pull request has been created, this launches
>>> more GitHub Actions
>>> > to run pre-commit tests.
>>> >
>>> > * Once the tests complete successfully and the changes
>>> have been approved
>>> > by the release manager, the pull request can me merged
>>> into the release branch.
>>> >
>>> > * After the pull request is merged, a GitHub Action
>>> automatically closes the
>>> > associated issue.
>>> >
>>> > Some Examples:
>>> >
>>> > Cherry-pick success:
>>> https://github.com/tstellar/llvm-project/issues/729
>>> <https://github.com/tstellar/llvm-project/issues/729>
>>> > Cherry-pick
>>> <https://github.com/tstellar/llvm-project/issues/729Cherry-pick
>>> <https://github.com/tstellar/llvm-project/issues/729Cherry-pick>>
>>> failure: https://github.com/tstellar/llvm-project/issues/730
>>> <https://github.com/tstellar/llvm-project/issues/730>
>>> <https://github.com/tstellar/llvm-project/issues/730
>>> <https://github.com/tstellar/llvm-project/issues/730>>
>>> > Manual Branch comment:
>>> https://github.com/tstellar/llvm-project/issues/710
>>> <https://github.com/tstellar/llvm-project/issues/710>
>>> <https://github.com/tstellar/llvm-project/issues/710
>>> <https://github.com/tstellar/llvm-project/issues/710>>
>>> >
>>> >
>>> > # Motivation
>>> >
>>> > Why do this? The goal is to make the release process
>>> more efficient and transparent.
>>> > With this new workflow, users can get automatic and
>>> immediate feedback when a commit
>>> > they want backported doesn't apply cleanly or introduces
>>> some test failures. With
>>> > the current process, these kinds of issues are
>>> communicated by the release manager,
>>> > and it can be days or even weeks before a problem is
>>> discovered and communicated back
>>> > to the users.
>>> >
>>> > Another advantage of this workflow is it introduces
>>> pre-commit CI to the release branch,
>>> > which is important for the stability of the branch and
>>> the releases, but also gives
>>> > the project an opportunity to experiment with new CI
>>> workflows in a way that
>>> > does not disrupt development on the main branch.
>>> >
>>> > # Implementation
>>> >
>>> > If this proposal is accepted, I would plan to implement
>>> this for the LLVM 14 release cycle based
>>> > on the following proof of concept that I have been
>>> testing for the last few releases:
>>> >
>>> >
>>> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml>
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml>>
>>>
>>> >
>>> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml>
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml>>
>>>
>>> >
>>> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml>
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml>>
>>>
>>> >
>>> > Thanks,
>>> > Tom
>>> >
>>> > _______________________________________________
>>> > LLVM Developers mailing list
>>> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>>> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
>>> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>>> <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>>
>>> >
>>>
>>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
More information about the llvm-dev
mailing list