[cfe-dev] [monorepo] Downstream branch zipping tool available

[Björn Pettersson A via cfe-dev]

Hi David.

Thanks for sharing your branch zipping migration script.


Unfortunately I think our situation is a little bit more complicated.

We have used llvm as the umbrella repo, so in llvm we have a "master"
branch (from the git single repo version of llvm) and a couple of
downstream branches (let's call them "down0", "down1") containing our
downstream work (with frequent merges from "master").
The downstream branches has tools/clang and runtimes/compiler-rt as
submodules, as well as a couple of downstream submodules.

In our downstream version of clang we have a similar structure.
A "master" branch (mapping to the git single repo version clang),
and a couple of downstream branches. The downstream branches has
tools/extra (i.e. clang-tools-extra) as a submodule.

I can also mention that the clang, compiler-rt and clang-tools-extra
submodules aren't present from the beginning of history. They have
been added later on.


I doubt that zip-downstream-fork.py will work out-of-the-box.
Hopefully I'll be able to patch it for our scenario. Any guidelines
might be helpful. But maybe it isn't even worth trying to adapt
zip-downstream-fork.py to do something useful for our scenario?


If someone else got a similar scenario, let me know. Perhaps we can
do some joint effort in adapting the zipper script.

Regards,
Björn



> -----Original Message-----
> From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of David
> Greene via llvm-dev
> Sent: den 12 november 2018 22:27
> To: llvm-dev at lists.llvm.org; cfe-dev at lists.llvm.org; libcxx-
> dev at lists.llvm.org; lldb-dev at lists.llvm.org; openmp-dev at lists.llvm.org;
> libclc-dev at lists.llvm.org; clangd-dev at lists.llvm.org
> Subject: [llvm-dev] [monorepo] Downstream branch zipping tool available
> 
> Building on the great work that James Knight did on
> migrate-downstream-fork.py (Thanks, James!) [1], I've created a simple
> tool to take migrated downstream fork branches and zip them into a
> single history given a history containing submodule updates of
> subprojects [2].
> 
> With migrate-downstream-fork.py, one is left with a set of unrelated
> histories, one per subproject:
> 
> llvm                             clang                       compiler-rt
> * V Add my fancy LLVM feature    * G Fix my dumb clang bug   * Z Merge
> from upstream compiler-rt
> 
> One can do an octopus merge to unify them:
> 
>   *-- Merge llvm, clang and compiler-rt
>   |\ \
>   * \ \  V Add my fancy LLVM feature
>   |  * |  G Fix my dumb clang bug
>   |  | *  Z Merge from upstream compiler-rt
> 
> Unfortunately, that doesn't show the logical history of development,
> where changes were effectively applied to subprojects in a linear
> fashion.  This makes it more difficult to do bisects, among other things
> because none of the downstream integration happens until the octopus
> merge.
> 
> Let's say that downstream you have a local mirror for each LLVM
> subproject you work on.  Suppose also that you have an "umbrella"
> repository that holds submodule references to all those local mirrors.
> Various commits in the umbrella update submodule references:
> 
>   * Update llvm submodule to V
>   * Update clang submodule to G
>   * Don't update any submodules, fix scripts or something
>   * Update compiler-rt submodule to Z
>   |
> 
> zip-downstream-fork.py will take these submodule updates and "inline"
> them into the umbrella history, making it appear that the downstream
> commits were applied against the monorepo in the order implied by the
> umbrella history:
> 
>   * A Add my fancy LLVM feature
>   * B Fix my dumb clang bug
>   * C Merge from upstream compiler-rt
>   |
> 
> Parent relationships for merges from upstream are preserved, though as
> top-level comments in zip-downstream-fork.py explain, the history graph
> can look a little strange.  Commits that don't update submodules are
> skipped on the assumption that they modify things uninteresting to a
> monorepo history.  Such commits could be preserved but doing so has some
> caveats as explained in the comments.  Perhaps your umbrella repository
> holds your build scripts.  You'd probably want to migrate that to the
> zipped history.  If there's strong demand for this I could look into
> doing it.
> 
> There are various other limitations to the tool explained in the
> comments.  It was enough to get us going and I'm hopeful it will be
> useful for others.  It seems to do the right thing with our repositories
> but YMMV.  Feel free to open PRs with bug fixes.  :)
> 
> To get this to work, you'll need to apply a PR for
> migrate-downstream-fork.py to fix issues with --revmap-out [3].
> 
>                         -David
> 
> [1] https://github.com/jyknight/llvm-git-migration/blob/master/migrate-
> downstream-fork.py
> [2] https://github.com/jyknight/llvm-git-
> migration/pull/2/commits/a3b44a294c20f1762cb42b5794e6130c5b27f22d
> [3] https://github.com/jyknight/llvm-git-migration/pull/1
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev