<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>As promised, here's a tool for migrating downstream forks of the split subproject repositories (e.g. <a href="https://github.com/llvm-mirror/**">https://github.com/llvm-mirror/**</a>) into a monorepo. It could be extended to handle conversion of a previous monorepo into the current one, too, but it doesn't currently.</div><div><br></div><div>(BTW -- just a reminder that the "llvm-git-prototype" repositories are NOT FINAL yet and may still change. Until it is finalized, please don't make repositories based on it, other than for demonstration/testing purposes.)</div><div><br></div><div>To use this tool, you must first pull together a git repository which includes the upstream monorepo, the relevant upstream split repositories, and any number of your private split repositories. Then, run the script to rewrite all the specified branches (destructively, in-place!), as if you'd made the commits on top of the monorepo.</div><div><br></div><div>Note -- this does not _interleave_ commits made to different subproject forks/branches. E.g., If you start with a branch of the "llvm" subproject repo and a branch of the "clang" subproject repo, after running this, you'll still have two branches. But they'll now both be based on a single upstream repository, and one will have your changes to the llvm/ subdir, and one will have your changes to the "clang" subdir. (if you wish to merge them afterwards, you may)</div><div><br></div><div>A known issue is that the script will give up if given a commit-graph which merges from multiple upstream svn branches, as that could cause conflicts in subprojects other than the one you've forked and resolved in the merge commit within. I'm not sure if anyone will run into that, but if they do, some heuristics can likely be added to handle it (e.g., throw out the old release branch's changes, just keep the new).<br></div><div><br></div><div>The tool is available here:</div><div><a href="https://github.com/jyknight/llvm-git-migration/blob/master/migrate-downstream-fork.py">https://github.com/jyknight/llvm-git-migration/blob/master/migrate-downstream-fork.py</a><br></div><div><br></div><div>I've tested it on the CHERI project, and it took about a minute to run 'migrate-downstream-fork' itself. (Pulling all the various source repositories took longer, of course.)</div><div><br></div><div>This result is here, for now (again, only for demonstration/testing!): <a href="https://github.com/jyknight/CHERI-monorepo-prototype">https://github.com/jyknight/CHERI-monorepo-prototype</a></div><br class="gmail-Apple-interchange-newline"><div>To that, I've only uploaded a single branch, "cheri", made from git merge on 3 of cheri's subproject repositories (llvm, clang, lld) after the migration. The other subprojects that CHERI forked have not been pulled up to a consistent revision in a long time, so can't be merged into a consistent target branch. I didn't look into any of the other branches in their repositories.</div>
<div><br></div><div>But, all branches of all of them did get migrated by the tool, and could be uploaded (either as-is, to their own branches, or after making appropriate merges between the subproject branches, whichever is preferable.)<br></div><div><br></div><div>Here's what I ran:</div><div><br>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">#!/bin/sh</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">set -exu</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">mkdir cheri-monorepo</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">cd cheri-monorepo</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">git init</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">git remote add monorepo <a href="https://github.com/llvm-git-prototype/llvm.git">https://github.com/llvm-git-prototype/llvm.git</a></span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">for x in llvm lld clang lldb libunwind libcxx; do</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>git remote add split/$x <a href="https://github.com/llvm-mirror/$x">https://github.com/llvm-mirror/$x</a></span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>git remote add cheri/$x <a href="https://github.com/CTSRD-CHERI/$x">https://github.com/CTSRD-CHERI/$x</a></span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">done</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">git fetch --all</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">../llvm-git-migration/migrate-downstream-fork.py refs/remotes/cheri refs/tags</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">git checkout -b cheri $(git merge-base refs/remotes/monorepo/master refs/remotes/cheri/llvm/master)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">git merge refs/remotes/cheri/{llvm,clang,lld}/master</span></p></div><div><br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Nov 5, 2018 at 1:53 PM David Greene <<a href="mailto:dag@cray.com">dag@cray.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Well shoot, you beat me to it. :)<br>
<br>
I've been working on a similar tool but it's not ready yet. Looking<br>
forward to trying yours!<br>
<br>
-David<br>
<br>
James Y Knight via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> writes:<br>
<br>
> I'm about to post exactly this tool -- I've been testing it on the<br>
> CHERI forks of llvm/clang/lld (lots of history and merges and stuff<br>
> there, makes a pretty nice test case!)<br>
><br>
> On Mon, Nov 5, 2018 at 1:07 PM David Greene via llvm-dev<br>
> <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br>
><br>
> Mehdi AMINI <<a href="mailto:joker.eph@gmail.com" target="_blank">joker.eph@gmail.com</a>> writes:<br>
> <br>
> > Yes, but that's the case for the zipper repo anyway: one merge<br>
> per<br>
> > commit. The point is that the second commit is just a trivial<br>
> merge,<br>
> > it wouldn't show up in a file `git log` for example.<br>
> > In the linear rewritten monorepo, adding the history taken from<br>
> the<br>
> > existing git mirror would lead to duplicated commits, as in<br>
> > *identical* commit / commit with the same diff but different git<br>
> > hashes. I'd expect git log to show us the two commits in the git<br>
> log<br>
> > of a single file.<br>
> <br>
> Would it be valuable to have a tool to take branches from existing<br>
> git<br>
> mirrors and rewrite them in terms of the monorepo so there would<br>
> be no<br>
> duplicate commits and everything would appear to have been done<br>
> against<br>
> the monorepo?<br>
> <br>
> I know Justin is worried about hashes in old e-mails being<br>
> invalid, but<br>
> the tool could include the mapping from old hash to new hash in<br>
> the<br>
> commit message. Of course that would only be done for local<br>
> downstream<br>
> commits, as the monorepo commits were already rewritten without<br>
> including that information. Would it be helpful to have the<br>
> monorepo<br>
> commits contain that information?<br>
> <br>
> -David<br>
> _______________________________________________<br>
> LLVM Developers mailing list<br>
> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
><br>
> _______________________________________________<br>
> LLVM Developers mailing list<br>
> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div></div>