[llvm-dev] Git Move: GitHub+modules proposal

[Alexey Denisov via llvm-dev]

Hello there,

Renato, thank you for putting everything together.

Talking about second question (commits mailing list): github provides set
of various web hooks. I think here we are interested
In 'push'es particularly.

Besides that it has some CI related integrations: buildbots can update pull
request status to show if tests are passing or not. The builds can be also
triggered using web hooks (issue_comment with specific text). IIRC swift
and rust do this (and more) in a very similar way.

Cheers,
Alex.

[1] https://developer.github.com/webhooks
<https://developer.github.com/webhooks/#payloads>

On Sunday, 26 June 2016, Renato Golin via llvm-dev <llvm-dev at lists.llvm.org>
wrote:

> So,
>
> It's been a while and the GitHub thread is officially dead, so I'll
> propose a development methodology based on the feedback from that
> thread. This is not *my* view, but all that was discussed in the
> threads.
>
> My objective is to form an official proposal to use Git as our main
> repository, overcoming all the problems we currently have without
> creating many others. In the end, I think Tanya wanted to make a vote,
> and see how string the community feels about it. The vote should be
> "Should we move to GitHub as the proposed workflow, or should we try
> to find another solution away from our own hosting?".
>
> The important part is that we *must* move from the current schema. It
> does not scale and the administrative costs are not worth the trouble.
> So, if we don't go with GitHub, we have to find another way out.
>
> The proposal
> ==========
>
> Move to GitHub with N+1 projects: all current LLVM projects + the
> "llvm-projs" umbrella. The latter will have all other projects as
> "submodules" with the intent to:
>
>  1. Control the history via server hooks updating a unique and
> auto-increment identifier, which will apply to every commit on its
> submodules (ie, every other LLVM project).
>  2. Serve as a reference for releases, buildbots, etc., checking out
> only the necessary submodules for each.
>  3. Have additional logic to handle the additional complexity for
> mailing lists, tools, buildbots to deal with the umbrella project
> *only*.
>
> The existing LLVM projects (llvm, clang, compiler-rt, etc) will
> continue on their own repositories and be built locally just like they
> are today. You can check them out individually inside the final
> directory (llvm/tools/clang) or use symbolic links, just like we do
> today. You can also checkout "llvm-projs" and update only the required
> submodules, and use symbolic links.
>
> The llvm-projs umbrella will have its own versioning, and tools can
> report that ID as their "version", if they're not in a release branch.
>
> Release branches should be off of master and have a linear history,
> just like master, in the exact same way we do now with SVN. This will
> guarantee the umbrella project will be able to correctly
> auto-increment the ID and make sure current tools work as usual.
>
> We don't want private branches to end up in upstream LLVM (only
> upstream release branches), but that's perfectly natural in GitHub,
> where anyone can fork and implement their features and own releases
> off of the upstream official repositories.
>
> This can work as well for upstream development of "feature branches",
> where upstream developers contribute to both repositories, but keeping
> a specific feature in test separate. Merges will still have to be like
> it is today, one patch at a time, or risk reverting the whole merge
> window if the buildbots start breaking, which can be impossible if the
> window is large or two or more windows get committed at the same time.
>
> For "feature branches" we could use git-imerge, but that's for the
> future and not considered in the first stage of the move.
>
> Git Submodules
> ---------------------
>
> There were concerns is submodules would work with our flow, but the
> concerns were addressed by demonstrating that:
>
>  1. Submodules can work in an umbrella project, which controls the
> auto-increment ID
>  2. You can check-out individual modules or all, so work well for
> releases and buildbots
>  3. The history is shared with the root project, so git-bisect works
> out of the box
>
> The Alternatives
> ---------------------
>
> A few alternatives were proposed, but git submodules ended up being
> considered more thoroughly. Here are some of the reasons:
>
>  * Google repo:
>
> It's an independent tool, which may suite us today, but not
> necessarily tomorrow. It may work well with the infrastructure that is
> already set for it on other projects (mostly Google projects like
> Android), but it does require some tooling (like git submodules). The
> point is, that it's much more likely to exist tooling for official git
> features than third-party projects, especially on Windows.
>
>  * All-in-one:
>
> Proposals to have all projects inside one big repo were quickly
> dismissed due to the problems it creates to *users* of LLVM as a
> library, and to build systems (specific buildbots) that don't need to
> monitor all changes all the time. It simply won't scale.
>
>  * Multiple clones:
>
> Allowing the projects to *exist* in different conditions (clang inside
> LLVM, or as a stand-alone library) will not scale. CMake will have to
> cope with all the different styles, it doesn't solve the unique
> auto-increment ID nor it helps downstream projects migrate to a common
> infrastructure.
>
> Questions
> ========
>
> In order to make this proposal final, I still need a few questions to
> be answered.
>
> 1. How will the umbrella project's auto-increment hook work?
>
> Will it be one ID for every commit in every other repo? How will it
> know which one came first? Does it matter? If we have two commit "at
> the same time", do we create a priority list?
>
> Ex. LLVM commits get a lower ID than Clang ones, because it's more
> likely that an LLVM commit needs to go in first.
>
> 2. How do we update the commits mailing lists?
>
> Can we add a mail script to the auto-increment ID hook? Or should we
> have a cron job that picks the new commits every 5 minutes in a server
> somewhere and email them (in ID order) to the respective lists?
>
> Approval
> =======
>
> Right now, we should not discuss if moving to Git or GitHub is a good
> idea or not. This is about the proposal itself. So, if you don't want
> Git or GitHub, wait for the voting to express that.
>
> If you do want Git and GitHub, than please keep your comments on
> topic, answer the questions and let's make sure we have a solid
> proposal that most Git proponents are happy with.
>
> If there is an alternative proposal (say, Google's repo), than this
> has to be separate, and well explained and accepted to be voted, too.
>
> Once we all agree in general, we should put it to vote. If there's an
> overwhelming majority (not sure how to measure this), and no critical
> problems (for example, learning new tools is less critical than
> breaking all buildbots), we should go with the move.
>
> For logistical reasons, if we do decide to move, I would like to do so
> before 3.10 / 4.0 branches.
>
> cheers,
> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <javascript:;>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160627/a6665c94/attachment.html>