<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">On Mon, Oct 15, 2018 at 7:58 AM Danila Malyutin <<a href="mailto:danila.malyutin@synopsys.com" target="_blank">danila.malyutin@synopsys.com</a>> wrote:<br></div><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_8976261634261660527gmail-m_-9164445157332819757WordSection1">
<p class="MsoNormal">Great to hear!<br>
<br>
Are there some migrations strategies for those who are using llvm-mirror repositories?<u></u><u></u></p>
<p class="MsoNormal"></p></div></div></blockquote><div><br></div><div><div dir="ltr">It'd be great if someone wrote down some exact step-by-step instructions for this. :)</div><div dir="ltr"><br></div><div dir="ltr">But, basically -- if you're migrating from another monorepo repository with the same layout, but different hashes, I find the easiest procedure is to simply add the new upstream into your existing repository and then rebase your branches onto the new upstream.<div><br></div><div>An alternative scheme, which has the advantage of also working if you're converting from separate repositories to a monorepo, is to run something like "git format-patch --stdout origin/master..mybranch > mybranch.patch", for each of your branches you want to save, and then, in a new repo, create new branches, and use "git am" to apply your patch. If you're converting from non-monorepo, then you can use the "--directory" argument to "git am" to apply the patch onto a subdirectory of the repository.</div><br class="gmail-Apple-interchange-newline"></div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_8976261634261660527gmail-m_-9164445157332819757WordSection1"><p class="MsoNormal">I’ve noticed that some projects in llvm git prototype not only have different commit SHAs (which is expected, since the commit messages are slightly different due to llvm-svn=N), but also different histories (first lnt commit in llvm-mirror
is 2 years older than in prototype).<u></u><u></u></p>
<p class="MsoNormal"><u></u></p></div></div></blockquote><div><br>I hadn't noticed that difference in the lnt history before -- looking into it now, I see that this is due to the initial checkin of /lnt/trunk being an svn copy of /zorg/trunk/lnt.</div><div><br></div><div>Because git-svn effectively runs "svn log llvm-project/lnt/trunk" to populate its copy, and that follows the history of trunk across the initial copy-from, that migration gets the history from before the copy. The new conversion uses path-based filtering of the entire tree, rather than following one path's log.</div><div><br></div><div>LNT is the only project which has this particular property. My inclination is to leave this as is -- to leave the previous history in the zorg repository. But it would also be possible to remove zorg/trunk/lnt from the zorg repository history, and instead put it into the LNT git history.<br></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_8976261634261660527gmail-m_-9164445157332819757WordSection1"><p class="MsoNormal"><b>From:</b> llvm-dev [mailto:<a href="mailto:llvm-dev-bounces@lists.llvm.org" target="_blank">llvm-dev-bounces@lists.llvm.org</a>]
<b>On Behalf Of </b>James Y Knight via llvm-dev</p></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_8976261634261660527gmail-m_-9164445157332819757WordSection1"><p class="MsoNormal">
<b>Sent:</b> Friday, October 12, 2018 01:28<br>
<b>To:</b> llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
<b>Subject:</b> [llvm-dev] New LLVM git repository conversion prototype<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">TLDR: <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_llvm-2Dgit-2Dprototype_&d=DwMFaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=YgdxWMcdqQPlU9EdetI-xI79G7ouw9_Us0dFsZnFQYU&m=N77kJ4LaWGbhMLDMsmsaWXefS9f6W0IZTDHibv5dCkc&s=K66CatS6_rX7Z84fG40bYi-BFdj1EH-Ge7BuasqHwaQ&e=" target="_blank">https://github.com/llvm-git-prototype/</a> exists
as a read-only mirror of SVN, and is being updated continuously with a script running on an llvm-project AWS VM.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><br>
Let me know what you think.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I had meant to get this prototype finalized 6 months ago, and I must apologize for the delay. I hope this is close to final for what we want our git repository to look like, and that we can move forward with the remainder of the work to
convert to git.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">At this point, there's no guarantee that the repository won't be rebuilt from scratch with new hashes, if some problem is discovered which requires changing something way back in history. But I hope we're now close to being able to declare
a conversion final -- and let people start depending on the hashes being stable.<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">This conversion uses the "flat monorepo" layout, like the previous existing git monorepo, and as discussed previously. The process generating it is different, which allows a more faithful conversion, including branches. I've also converted
a bunch of the auxiliary repositories.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I would request that other people help take charge of the remainder of the work. Most importantly -- making a plan for implementing the *rest* of the migration. We have
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__llvm.org_docs_Proposals_GitHubMove.html&d=DwMFaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=YgdxWMcdqQPlU9EdetI-xI79G7ouw9_Us0dFsZnFQYU&m=N77kJ4LaWGbhMLDMsmsaWXefS9f6W0IZTDHibv5dCkc&s=qS4wDBMYZgiJajPClPRreKzuXJVsb2635t3auD50xhk&e=" target="_blank">
https://llvm.org/docs/Proposals/GitHubMove.html</a>, but I think it'll need significant fleshing out and updating. I'm happy to assist with the rest of the migration, but I'd like to _not_ be primarily responsible for other parts beyond svn->git repository
conversion.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Some things that could be discussed in such a plan:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * Verifying that this conversion is good, what we want, and declaring it final (at which point the hashes can be relied upon not to change).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * Any particular steps wanted here?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * Converting buildbots to use git.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * Phabricator changes?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * How do email notifications get sent for commits?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * Gathering github accounts for all committers, adding them to a github team.<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"> * Deciding upon and announcing a timeline for switching over.<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> * Proposing, implementing, and testing new workflows for direct git usage:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * Github pull requests instead of (or in addition to?) phabricator?<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"> * Github Protected Branch configuration options?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * E.g. -- direct pushing to git without any restriction, or, require that pull requests be created first?<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> * Automated Pre-commit testing? Do we setup CI (e.g. <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__travis-2Dci.org&d=DwMFaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=YgdxWMcdqQPlU9EdetI-xI79G7ouw9_Us0dFsZnFQYU&m=N77kJ4LaWGbhMLDMsmsaWXefS9f6W0IZTDHibv5dCkc&s=DeqmBM4n_BBYesAelEv-fMXEE5koKo0ZQVco3Pt2kOI&e=" target="_blank">
travis-ci.org</a>) to do some testing on pull requests, to reduce avoidable tree breakages?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * Any other github configuration options that need to be decided upon?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * ....other things I forgot about at the moment...<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * Timeline for switchover.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Anyways, what's been done _so far_ is a full SVN->Git repository conversion. This conversion:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * Places the SVN revision number into the commit message, as "llvm-svn=1234"<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> * Automatically preserves all branches from the SVN repository (it merges the branches named /$project/branches/$name into a single "$name" branch, attempting, as much as possible, to make the branch-creation commits not look insane).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> * Attempts to convert the svn branches in the "tags" subdir into annotated git tags pointing to the proper commit on the parent branch, where feasible. Sometimes this is impossible, since the "tags" have had modifications after their
creation. (They're just branches in SVN, so you can do that, although you shouldn't). If so, they're preserved as a branch named "svntag/$name", instead.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> * Preserves the svn id -> email mapping that was in-use at the time of each SVN commit, as far as is known.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> * Fixes a bunch of -- but not all -- the CVS->SVN conversion errors (due, e.g., to files being renamed directly in the CVS repository).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Most of the SVN directories are migrated into sub-directories inside the main "llvm" mono-repository:<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"> * cfe (renamed to clang in the conversion)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * clang-tools-extra<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * compiler-rt<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * debuginfo-tests<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * dragonegg (also "gcc-plugin", the original name)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * libclc<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * libcxx<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * libcxxabi<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * libunwind<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * lld<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * lldb<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * llgo<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * llvm<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * openmp<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * parallel-libs<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * polly<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * pstl<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * stacker (deleted after r40406)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">(Additionally, files added to the "monorepo-root/trunk" directory in SVN end up at the root of this repository).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Some SVN projects are still active, but not part of the LLVM codebase. These get migrated to their own separate git repositories:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * lnt<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * test-suite<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * www<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * www-pubs<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * www-releases ## TODO. Not done yet as it requires the use of git-lfs, due to large files.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * zorg<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">A couple inactive projects which are somewhat related to the LLVM codebase, migrated to separate repos:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * poolalloc<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * safecode<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Legacy projects that are not particularly interesting, migrated to a single separate git repository named "archive":<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * clang-tests # Copy of GCC 4.2 testsuite, modified to work with clang<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * clang-tests-external # Copy of GDB testsuite<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * llvm-gcc-4.0 # GCC 4.0, modified for llvm<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * llvm-gcc-4.2 # GCC 4.2, modified for llvm<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * llvm-gcc-4-2 # (merge with above)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * java<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * vmkit<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * nightly-test-server<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * llbrowse # An LLVM bitcode GUI browser<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * television # A different LLVM GUI browser; shows effects of transforms, etc<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * website # 2007-era snapshot of website, not actually maintained here.<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"> * core, llvm-top, sample, support, hlvm # from the "HLVM" refactoring attempt.<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Projects _not_ migrated from SVN in this conversion, since they're elsewhere already:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * giri # Never actually developed here; actually <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_liuml07_giri&d=DwMFaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=YgdxWMcdqQPlU9EdetI-xI79G7ouw9_Us0dFsZnFQYU&m=N77kJ4LaWGbhMLDMsmsaWXefS9f6W0IZTDHibv5dCkc&s=_CpoeuN6AI232B9eqd28X5dAI8MoTC6_t5vDWz8f_2c&e=" target="_blank">https://github.com/liuml07/giri</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> * klee # Already migrated to github with history; <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_klee_klee&d=DwMFaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=YgdxWMcdqQPlU9EdetI-xI79G7ouw9_Us0dFsZnFQYU&m=N77kJ4LaWGbhMLDMsmsaWXefS9f6W0IZTDHibv5dCkc&s=UTwnvwghVXONKHDJjon3lvCG3ZRJT-xZDJmAb3whxIw&e=" target="_blank">https://github.com/klee/klee</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <br><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div></div></div></div></div>