[llvm-dev] [RFC] Move py-mlir-release to new top-level repo in the LLVM org

[Tom Stellard via llvm-dev]

On 1/7/21 10:55 AM, Stella Laurenzo via llvm-dev wrote:
> Hi folks, I would like to propose that we create a new top-level repo in 
> the LLVM organization for organizing the Python MLIR Releases (both 
> daily and official numbered releases, whenever we are ready for such a 
> thing) and corresponding pushes to package repositories, etc.
> 

For those of use that are unfamiliar, can you explain what the "Python 
MLIR Releases" are?

> I have prototyped such a release process in a personal repo: 
> https://github.com/stellaraccident/mlir-py-release
> 
> Additional development on that release process is currently blocked on 
> more work on the shared library organization in LLVM (discussed here 
> https://lists.llvm.org/pipermail/llvm-dev/2021-January/147567.html and 
> being worked on independently) but it is useful as is and a reasonable 
> starting point for further work.
> 
> I would propose that we just fork my current repo into a new one in the 
> LLVM organization and then take the necessary steps to get 
> credentials/permissions/secrets set up in the new context.
> 
> Some answers to questions that may come up:
> 
>   * *Why should this be a repo separate from llvm-project? *These kind
>     of automation repos tend to have a lot of "garbage" commits that I
>     think is best if they do not pollute the main repo (and also don't
>     face contention on automatic jobs bumping things, etc). They also
>     tend to require special permissions and secrets that we will want to
>     more tightly control. They also make use of other GitHub features
>     that it seems like we would like not polluting the main development
>     flow ("Releases" tab, Actions, etc). Also, this is the kind of thing
>     that tends to get revised en-masse periodically, and again, it would
>     be good to not pollute the monorepo.

There really aren't many files in this repo, do you anticipate it 
growing significantly?

>   * *Why not land this in llvm-zorg? *llvm-zorg claims to be for "LLVM
>     Testing Infrastructure" and seems well scoped to that statement.
>     What I am managing above is periodic, automated release tooling
>     based on open-source CI systems (currently GitHub Actions), which
>     are fairly standardized across the Python releasing community, easy
>     to set up, etc.

llvm-zorg also handles generating the websites.  My personal opinion is 
that it would be OK to try to do this in llvm-zorg, but you're probably 
better off asking Galina about that.  I guess the downside of using 
llvm-zorg is you don't get the releases tab.

Why did you choose to write the checkout_repo.py script in python rather 
than using the GitHub checkout action, or writing your own custom action?

>   * *What ultimately will the code in this repo do?*
>       o Have periodic GitHub actions to select new LLVM revisions and
>         schedule daily/snapshot releases.

Do you have any idea of much of the GitHub actions resources this would 
use?  e.g. how many hours per week per Operating System?

>       o Have manual actions for triggering official, numbered releases.
>       o Facilities for building Python wheels for PyPi and house any
>         additional metadata/automation needed for anaconda.
>       o Builds releases for all supported operating systems (currently
>         Linux/CentOS7/manylinux2014, MacOS, and Windows) and supported
>         Python versions (currently 3.6, 3.7, 3.8, 3.9).
>       o Publish release artifacts on the Releases tab for daily/snapshot
>         releases.
>       o Provide a stable reference point for downstream projects that
>         extend MLIR-Python and need to produce version-matched artifacts
>         of their own.
>   * *Could this graduate to be more than "MLIR" python?* Maybe. I chose
>     the name because that is what I am focused on and didn't want to
>     grab too much land. But there is nothing stopping this from becoming
>     automation for general LLVM monorepo+incubator Python releasing.

I think it would be great to generalize this.  I would also like to 
automate parts the main LLVM release, and there seems to be some overlap 
with what you are doing.

-Tom

>   * *What if we don't do this?*
>       o *Option A:* We keep running this in a private repo with the
>         disclaimer that is currently at the top: "Note that this is a
>         prototype of a real MLIR release process being run by a member
>         of the community. These are not official releases of the LLVM
>         Foundation in any way, and they are likely only going to be
>         useful to people actually working on LLVM/MLIR until we get
>         things productionized." We would miss opportunities for
>         convergence with other projects and would cause things to fragment.
>       o *Option B: *We only publish Python bindings in official LLVM
>         release packages, and only for the Python version they are built
>         with. We don't release Python binaries through normal package
>         management channels.


> 
> Opinions?
> - Stella
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>