[cfe-dev] [llvm-dev] RFC: New Automated Release Workflow (using Issues and Pull Requests)

Philip Reames via cfe-dev cfe-dev at lists.llvm.org
Mon Dec 20 15:34:15 PST 2021


On 12/20/21 3:24 PM, Tom Stellard via llvm-dev wrote:
> On 12/20/21 09:16, Tom Stellard wrote:
>> On 12/18/21 15:04, David Blaikie wrote:
>>>
>>>
>>> On Fri, Dec 17, 2021 at 6:38 PM Tom Stellard <tstellar at redhat.com 
>>> <mailto:tstellar at redhat.com>> wrote:
>>>
>>>     On 12/17/21 16:47, David Blaikie wrote:
>>>      > Sounds pretty good to me - wouldn't mind knowing more about/a 
>>> good summary of the effects of this on project/repo/etc 
>>> notifications that Mehdi's mentioning. (be good to have a write up 
>>> of the expected impact/options to then discuss - from the thread so 
>>> far I understand some general/high level concerns, but it's not 
>>> clear to me exactly how it plays out)
>>>      >
>>>
>>>     The impact is really going to depend on the person and what 
>>> notification preferences they
>>>     have/want.  If you are already watching the repo with the 
>>> default settings, then you probably
>>>     won't notice much of a difference given the current volume of 
>>> notifications.
>>>
>>>
>>> I think I'm on the default settings - which does currently mean a 
>>> notification for every issue update, which is a lot. Given that 
>>> llvm-bugs at email.llvm.org <mailto:llvm-bugs at email.llvm.org> has been 
>>> re-enabled, sending mail only on issue creation, I & others might 
>>> opt back in to that behavior by disabling the baseline "notify on 
>>> everything" to "notify only on issues I'm mentioned in".
>>>
>>> I guess currently the only email that github is generating is one 
>>> email per issue update. We don't have any pull requests, so there 
>>> aren't any emails for that, yeah?
>>>
>>> So this new strategy might add a few more back-and-forth on each 
>>> cherrypick issue (for those using llvm-bugs & disabling general 
>>> issue notifications, this will not be relevant to them - there won't 
>>> be more issues created, just more comments on existing issues). But 
>>> there will be some more emails generated related to the pull 
>>> requests themselves, I guess? So each cherrypick goes from 2 emails 
>>> to llvm-bugs (the issue creation and closure) to, how many? 4 (2 for 
>>> llvm-bugs and I guess at least 2 for the pull request - one to make 
>>> the request and one to close it - maybe a couple more status ones 
>>> along the way?)
>>>
>>
>> I think the number of net new comments on issues will be very minimal 
>> or none at all.  The
>> automated comments that are created by this process are replacing 
>> comments that I'm already making
>> manually.
>>
>> So 2+ for pull requests is probably a good estimate.  I still need to 
>> figure out how many notifications
>> get generated for Actions with the default settings.
>>
>
> I did some research on the notifications and here is what I came up with:
>
> From what I can tell, notifications for actions are only sent to the
> user that initiated the event that led to the actions, so there would
> be no global notifications sent for the actions used by this workflow.
>
> There have been 131 bugs marked as release blockers in the llvm-13 cycle,
> this includes the 13.0.0 and 13.0.1 release.  In the best case scenario,
> this proposal would generate 2 additional notifications per issue
> (1 for creating a pull request and 1 for merging it), and 0 net new
> issue comments (the automated comments just replace manual comments).
>
> If you assume that no manual comments would be replaced by the 
> automation,
> then in the typical use case there would be a maximum of  4 notifications
> generated from issues (/cherry-pick comment, cherry-pick failed comment,
> /branch comment, /pull-request comment). In addition to the 2 pull
> request notifications.
>
> Based on this, my estimate is that this proposal will produce between
> (2 * 131) = 262 and (6 * 131) = 786 net new notifications every 6 months.
> Or between 1.46 and 4.367 net new notifications per day.
>
> For comparison, on Fri Dec 17, I received 115 email notifications from
> the llvm/llvm-project repo.
>
> The pull request emails should be easy for people to filter out of their
> inboxes with a rule.  Pull request emails would have llvm/llvm-project in
> the To: field and have '(PR #123456)' at the end of the Subject: field
> (where 123456 is pull request number).
>
> For people who filter out the pull request notifications, they would 
> have between
> 0 and 2.9 net new notifications per day.

This seems both fairly minimal, and well justified to me.

Philip

>
> - Tom
>
>
>> --Tom
>>
>>>     If people want to give their notification preferences, I can try 
>>> to look at how
>>>     this change will impact specific configurations.
>>>
>>>
>>> @Mehdi AMINI <mailto:joker.eph at gmail.com> - are there particular 
>>> scenarios you have in mind that'd be good to work through?
>>>
>>>
>>>     -Tom
>>>
>>>
>>>      > On Fri, Dec 17, 2021 at 1:15 PM Tom Stellard via llvm-dev 
>>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> 
>>> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>> 
>>> wrote:
>>>      >
>>>      >     Hi,
>>>      >
>>>      >     Here is a proposal for a new automated workflow for 
>>> managing parts of the release
>>>      >     process.  I've been experimenting with this over the past 
>>> few releases and
>>>      >     now that we have migrated to GitHub issues, it would be 
>>> possible for us to
>>>      >     implement this in the main repo.
>>>      >
>>>      >     The workflow is pretty straight forward, but it does use 
>>> pull requests.  My
>>>      >     idea is to enable pull requests for only this automated 
>>> workflow and not
>>>      >     for general development (i.e. We would still use 
>>> Phabricator for code review).
>>>      >     Let me know what you think about this:
>>>      >
>>>      >
>>>      >     # Workflow
>>>      >
>>>      >     * On an existing issue or a newly created issue, a user 
>>> who wants to backport
>>>      >     one or more commits to the release branch adds a comment:
>>>      >
>>>      >     /cherry-pick <commit_sha> <..>
>>>      >
>>>      >     * This starts a GitHub Action job that attempts to 
>>> cherry-pick the commit(s)
>>>      >     to the current release branch.
>>>      >
>>>      >     * If the commit(s) can be cherry-picked cleanly, then the 
>>> GitHub Action:
>>>      >           * Pushes the result of the cherry-pick to a branch 
>>> in the
>>>      >             llvmbot/llvm-project repo called issue<n>, where 
>>> n is the number of the
>>>      >             GitHub Issue that launched the Action.
>>>      >
>>>      >           * Adds this comment on the issue: /branch 
>>> llvmbot/llvm-project/issue<n>
>>>      >
>>>      >           * Creates a pull request from 
>>> llvmbot/llvm-project/issue<n> to
>>>      >             llvm/llvm-project/release/XX.x
>>>      >
>>>      >           * Adds a comment on the issue: /pull-request #<n>
>>>      >             where n is the number of the pull request.
>>>      >
>>>      >     * If the commit(s) can't be cherry-picked cleanly, then 
>>> the GitHub Action job adds
>>>      >     the release:cherry-pick-failed label to the issue and 
>>> adds a comment:
>>>      >     "Failed to cherry-pick <commit_sha> <..>" along with a 
>>> link to the failing
>>>      >     Action.
>>>      >
>>>      >     * If a user has manually cherry-picked the fixes, 
>>> resolved the conflicts, and
>>>      >     pushed the result to a branch on github, they can 
>>> automatically create a pull
>>>      >     request by adding this comment to an issue: /branch 
>>> <user>/<repo>/<branch>
>>>      >
>>>      >     * Once a pull request has been created, this launches 
>>> more GitHub Actions
>>>      >     to run pre-commit tests.
>>>      >
>>>      >     * Once the tests complete successfully and the changes 
>>> have been approved
>>>      >     by the release manager, the pull request can me merged 
>>> into the release branch.
>>>      >
>>>      >     * After the pull request is merged, a GitHub Action 
>>> automatically closes the
>>>      >     associated issue.
>>>      >
>>>      >     Some Examples:
>>>      >
>>>      >     Cherry-pick success: 
>>> https://github.com/tstellar/llvm-project/issues/729 
>>> <https://github.com/tstellar/llvm-project/issues/729>
>>>      >     Cherry-pick 
>>> <https://github.com/tstellar/llvm-project/issues/729Cherry-pick 
>>> <https://github.com/tstellar/llvm-project/issues/729Cherry-pick>> 
>>> failure: https://github.com/tstellar/llvm-project/issues/730 
>>> <https://github.com/tstellar/llvm-project/issues/730> 
>>> <https://github.com/tstellar/llvm-project/issues/730 
>>> <https://github.com/tstellar/llvm-project/issues/730>>
>>>      >     Manual Branch comment: 
>>> https://github.com/tstellar/llvm-project/issues/710 
>>> <https://github.com/tstellar/llvm-project/issues/710> 
>>> <https://github.com/tstellar/llvm-project/issues/710 
>>> <https://github.com/tstellar/llvm-project/issues/710>>
>>>      >
>>>      >
>>>      >     # Motivation
>>>      >
>>>      >     Why do this?  The goal is to make the release process 
>>> more efficient and transparent.
>>>      >     With this new workflow, users can get automatic and 
>>> immediate feedback when a commit
>>>      >     they want backported doesn't apply cleanly or introduces 
>>> some test failures.  With
>>>      >     the current process, these kinds of issues are 
>>> communicated by the release manager,
>>>      >     and it can be days or even weeks before a problem is 
>>> discovered and communicated back
>>>      >     to the users.
>>>      >
>>>      >     Another advantage of this workflow is it introduces 
>>> pre-commit CI to the release branch,
>>>      >     which is important for the stability of the branch and 
>>> the releases, but also gives
>>>      >     the project an opportunity to experiment with new CI 
>>> workflows in a way that
>>>      >     does not disrupt development on the main branch.
>>>      >
>>>      >     # Implementation
>>>      >
>>>      >     If this proposal is accepted, I would plan to implement 
>>> this for the LLVM 14 release cycle based
>>>      >     on the following proof of concept that I have been 
>>> testing for the last few releases:
>>>      >
>>>      > 
>>> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml 
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml> 
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml 
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml>> 
>>>
>>>      > 
>>> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml 
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml> 
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml 
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow-create-pr.yml>> 
>>>
>>>      > 
>>> https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml 
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml> 
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml 
>>> <https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml>> 
>>>
>>>      >
>>>      >     Thanks,
>>>      >     Tom
>>>      >
>>>      >     _______________________________________________
>>>      >     LLVM Developers mailing list
>>>      > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> 
>>> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
>>>      > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev 
>>> <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> 
>>> <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev 
>>> <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>>
>>>      >
>>>
>>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


More information about the cfe-dev mailing list